Optimized JVM memory usage and executor configuration because my Railway bill is beating me up!!! (Added a doc for this, see 2026-01-13-memory-optimization.md)

Author: timan-z

README.md

@@ -1700,6 +1700,13 @@ Planned enhancements to be added incrementally:
 
 These enhancements build directly on the existing metrics and APIs without requiring backend architectural changes.
 
+### Potential Changes Pre-CloudQueue
+
+#### Integrating Virtual Threads (Java 21)
+
+SpringQueuePro currently uses platform threads for clarity and debuggability. A future enhancement will migrate worker execution to **Java 21 virtual threads**, which dramatically reduce per-thread memory overhead and allow higher concurrency with minimal resource cost.
+- This change is intentionally deferred to keep the current architecture stable for demos and interviews.
+
 ---
 
 ### CloudQueue — AWS-Native Evolution

other-docs/2026-01-13-memory-optimization.md

@@ -0,0 +1,150 @@
+# SpringQueuePro — Memory Optimization (2026-01-13)
+
+This project is a massive memory hog and has been racking up my monthly Railway bill a ridiculous amount. This document is dedicated to a focused memory optimization pass performed on **2026-01-23** aimed at reducing JVM and application-level memory usage when running and hosting SpringQueuePro on any memory-billed PaaS platform. (As mentioned, I'm currently hosting this on Railway with their hobby plan which costs ~5CAD a month, which is the covered amount for your hosted project's memory usage. But this project, and even the base SpringQueue version, has been jumping my bill close to ~20CAD by month's end).
+
+## Memory Usage Breakdown
+
+It isn't *that* surprising that this project (**JVM + SpringBoot + metrics + Redis + GraphQL** being hosted **24/7**) has been so memory-hungry. The memory usage breakdown should look something like this:
+| Source                              | Memory Hog (Why)        |
+| ----------------------------------- | ------------------------------------------------ |
+| **JVM default heap sizing**         | JVM happily reserves hundreds of MB even at idle |
+| **Spring Boot auto-config**         | Loads frequently while not in use             |
+| **ExecutorServices + schedulers**   | Threads = stack memory + queues                  |
+| **Micrometer + metrics registries** | Counters, timers, tags accumulate                |
+| **Redis + caches + in-memory maps** | authoritative + cached state            |
+| **GraphQL + GraphiQL**              | Extra schema + reflection + servlet overhead     |
+
+## Steps Taken to Optimize Memory Usage
+
+### 1. Capping the JVM heap on Railway
+
+Added the following environmental variable on Railway:
+```
+JAVA_TOOL_OPTIONS=-Xms64m -Xmx256m -XX:+UseG1GC -XX:MaxMetaspaceSize=128m
+```
+An explicit limit on the JVM's memory allocation w/ the heap and metaspace. This will prevent over-reservation and reducing idle memory use.
+
+---
+
+### 2. Disable GraphiQL in Production
+
+Adding this to `application-prod.yml`:
+```yaml
+spring:
+  graphql:
+    graphiql:
+      enabled: false
+```
+GraphiQL loads static assets, schema introspection, and additional servlet mappings that are unnecessary in production. Disabling it reduces baseline memory usage and class loading overhead.
+
+---
+
+### 3. Limiting Actuator & Micrometer Exposure
+Original version of what's shown below included health, metrics, and prometheus in **include**:
+```yaml
+management:
+  endpoints:
+    web:
+      exposure:
+        include: health,prometheus
+  metrics:
+    enable:
+      jvm: false
+      process: false
+      system: false
+      executor: false
+      hibernate: false
+      logback: false
+```
+This disables high-cardinality, always-on metric groups (JVM, system, Hibernate, etc.) that consume memory continuously, while preserving the `/prometheus` endpoint needed for future Grafana integration.
+
+---
+
+### 4. Remove Legacy In-Memory Task Queue
+I originally had these fields and methods marked as `@Deprecated` but unfortunately these still consume memory even if they're not in use.
+
+- Removed:
+
+  ```java
+  private final ConcurrentHashMap<String, Task> jobs;
+  ```
+- Removed deprecated methods such as:
+
+  ```java
+  @Deprecated
+  public List<Task> getJobs() {
+      return new ArrayList<>(jobs.values());
+  }
+  ```
+
+The legacy in-memory task map retained domain objects and enabled heap copying under load. Removing it eliminates unnecessary object retention and reinforces PostgreSQL as the single source of truth.
+
+---
+
+### 5. Cap Worker and Scheduler Threads via Configuration
+Setting this explicit configuration in `application-prod.yml` for my `QueueProperties.java` file:
+```yaml
+queue:
+  main-exec-worker-count: 5
+  sched-exec-worker-count: 2
+```
+Thread stacks consume ~1MB each by default. Explicitly capping worker and scheduler counts prevents unbounded thread creation while still allowing meaningful concurrency for demos and testing.
+
+---
+
+### 6. Removing Unnecessary Dependencies (pom.xml)
+
+Getting rid of stuff like this that was lying around:
+```xml
+<artifactId>spring-boot-starter-webflux</artifactId>
+<scope>test</scope>
+```
+*For this example specifically*, WebFlux pulls in Reactor and Netty, increasing classpath scanning, memory usage, and startup overhead even when unused.
+
+---
+
+### 7. Replace Executor Factories with Explicit ThreadPoolExecutor
+
+Making this change in `ExecutorConfig.java` (*the original code is what's commented out in the snippet*):
+
+```java
+@Bean("execService")
+public ExecutorService taskExecutor() {
+    /*return Executors.newFixedThreadPool(props.getMainExecWorkerCount(), r -> {
+            Thread t = new Thread(r);
+            t.setName("QS-Worker-" + t.getId());
+            return t;
+    });*/
+    return new ThreadPoolExecutor(
+            props.getMainExecWorkerCount(),
+            props.getMainExecWorkerCount(),
+            0L,
+            TimeUnit.MILLISECONDS,
+            new LinkedBlockingQueue<>(1000),
+            r -> {
+                Thread t = new Thread(r);
+                t.setName("QS-Worker-" + t.getId());
+                t.setDaemon(true);
+                return t;
+            }
+    );
+}
+```
+
+Using an explicit `ThreadPoolExecutor` avoids unbounded task queues, enables backpressure under load, and reduces memory spikes during stress testing.
+
+---
+
+### 8. Remove Legacy Metrics
+
+Removed counters and gauges tied to deprecated code paths (e.g., legacy in-memory queue metrics). Got rid of stuff like (which relates to outdated, deprecated code):
+```java
+@Bean
+    public Gauge inMemoryQueueSizeGauge(MeterRegistry registry, QueueService queueService) {
+        return Gauge.builder("springqpro_queue_memory_size", queueService, q -> q.getJobMapCount())
+                .description("Number of tasks currently in legacy in-memory queue")
+                .register(registry);
+}
+```
+
+Legacy metrics retained references to unused services and data structures, increasing object retention and complicating the runtime memory graph.

springqpro-backend/pom.xml

@@ -156,12 +156,12 @@
             <scope>test</scope>
         </dependency>
 
-        <dependency>
+        <!--<dependency>
             <groupId>org.springframework.boot</groupId>
             <artifactId>spring-boot-starter-webflux</artifactId>
             <version>3.5.7</version>
             <scope>test</scope>
-        </dependency>
+        </dependency>-->
 
         <dependency>
             <groupId>org.apache.commons</groupId>

springqpro-backend/src/main/java/com/springqprobackend/springqpro/config/ExecutorConfig.java

@@ -3,9 +3,7 @@
 import org.springframework.context.annotation.Bean;
 import org.springframework.context.annotation.Configuration;
 
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.*;
 
 /* ExecutorConfig.java
 --------------------------------------------------------------------------------------------------
@@ -52,11 +50,24 @@ public class ExecutorConfig {
 
     @Bean("execService")
     public ExecutorService taskExecutor() {
-       return Executors.newFixedThreadPool(props.getMainExecWorkerCount(), r -> {
-           Thread t = new Thread(r);
-           t.setName("QS-Worker-" + t.getId());
-           return t;
-       });
+        /*return Executors.newFixedThreadPool(props.getMainExecWorkerCount(), r -> {
+               Thread t = new Thread(r);
+               t.setName("QS-Worker-" + t.getId());
+               return t;
+        });*/
+        return new ThreadPoolExecutor(
+                props.getMainExecWorkerCount(),
+                props.getMainExecWorkerCount(),
+                0L,
+                TimeUnit.MILLISECONDS,
+                new LinkedBlockingQueue<>(1000),
+                r -> {
+                    Thread t = new Thread(r);
+                    t.setName("QS-Worker-" + t.getId());
+                    t.setDaemon(true);
+                    return t;
+                }
+        );
     }
 
     @Bean("schedExec")

springqpro-backend/src/main/java/com/springqprobackend/springqpro/config/ProcessingMetricsConfig.java

@@ -8,6 +8,8 @@
 import org.springframework.context.annotation.Bean;
 import org.springframework.context.annotation.Configuration;
 
+import java.time.Duration;
+
 @Configuration
 public class ProcessingMetricsConfig {
     @Bean
@@ -44,8 +46,9 @@ public Counter tasksRetriedCounter(MeterRegistry registry) {
     public Timer processingTimer(MeterRegistry registry) {
         return Timer.builder("springqpro_task_processing_duration")
                 .description("Time spent executing task handlers")
-                .publishPercentiles(0.50, 0.90, 0.95, 0.99)
+                .publishPercentiles(0.50, 0.90, 0.95)
                 .publishPercentileHistogram()
+                .sla(Duration.ofMillis(100), Duration.ofMillis(500), Duration.ofSeconds(1))
                 .register(registry);
     }
     // The one below is for the number of tasks made by users sending GraphQL queries:
@@ -55,22 +58,24 @@ public Counter apiTaskCreateCounter(MeterRegistry registry) {
                 .description("Tasks created from GraphQL API")
                 .register(registry);
     }   // TO-DO: GraphQL is my main API thing, but I'm keeping the REST stuff too -- maybe add another Counter for that specifically?
-    @Bean
+    /*@Bean
     public Counter queueEnqueueCounter(MeterRegistry registry) {
         return Counter.builder("springqpro_queue_enqueue_total")
                 .description("In-memory enqueue() calls (legacy path)")
                 .register(registry);
-    }   // <-- DEBUG: I can't remember if this is even used anymore at this point in the program...
+    }*/   // <-- DEBUG: I can't remember if this is even used anymore at this point in the program...
+    // 2026-01-13-NOTE: ^ Removed because the jobs field is a memory hog even if it's deprecated and it's racking up my Railway bill.
     @Bean
     public Counter queueEnqueueByIdCounter(MeterRegistry registry) {
         return Counter.builder("springqpro_queue_enqueue_by_id_total")
                 .description("enqueueById() calls feeding into ProcessingService")
                 .register(registry);
     }
-    @Bean
+    /*@Bean
     public Gauge inMemoryQueueSizeGauge(MeterRegistry registry, QueueService queueService) {
         return Gauge.builder("springqpro_queue_memory_size", queueService, q -> q.getJobMapCount())
                 .description("Number of tasks currently in legacy in-memory queue")
                 .register(registry);
-    }   // <-- DEBUG: I can't remember if this is even used anymore at this point in the program...
+    }*/   // <-- DEBUG: I can't remember if this is even used anymore at this point in the program...
+    // 2026-01-13-NOTE: ^ Removed because the jobs field is a memory hog even if it's deprecated and it's racking up my Railway bill.
 }

springqpro-backend/src/main/java/com/springqprobackend/springqpro/controller/rest/ProducerController.java

@@ -89,27 +89,27 @@ public ResponseEntity<Map<String, String>> handleEnqueue(@Valid @RequestBody Enq
 
     // 2. The equivalent of GoQueue's "http.HandleFunc("/api/jobs", func(w http.ResponseWriter, r *http.Request) {...}" function:
     // From producer.go: "THIS IS FOR [GET /api/jobs] and [GET /api/jobs?status=queued]" <-- hence why we're using @RequestParam
-    @GetMapping("/jobs")
+    /*@GetMapping("/jobs")
     public ResponseEntity<List<Task>> handleListJobs(@RequestParam(required = false) String status) {
         //Task[] allJobs = queue.getJobs();
         //List<Task> filtered = Arrays.stream(allJobs).filter(t -> t != null && (status == null || t.getStatus().toString().equalsIgnoreCase(status))).collect(Collectors.toList());
         List<Task> allJobs = queue.getJobs();
         logger.info("The value of allJobs is {} and the value of queue.getJobs() is {}", allJobs, queue.getJobs());
         List<Task> filtered = allJobs.stream().filter(t -> t != null && (status == null || t.getStatus().toString().equalsIgnoreCase(status))).collect(Collectors.toList());
         return ResponseEntity.ok(filtered);
-    }
+    }*/
 
     // The handlers below will be for the individual methods in GoQueue's "http.HandleFunc("/api/jobs/", func(w http.ResponseWriter, r *http.Request) {...}" function:
     // 3. This is for [GET /api/jobs/:id]:
-    @GetMapping("/jobs/{id}")
+    /*@GetMapping("/jobs/{id}")
     public ResponseEntity<Task> handleGetJobById(@PathVariable String id) {
         Task t = queue.getJobById(id);
         if(t == null) return ResponseEntity.notFound().build();
         return ResponseEntity.ok(t);
-    }
+    }*/
 
     // 4. This is for [POST /api/jobs/:id/retry]:
-    @PostMapping("/jobs/{id}/retry")
+    /*@PostMapping("/jobs/{id}/retry")
     public ResponseEntity<?> handleRetryJobById(@PathVariable String id) {
         Task t = queue.getJobById(id);
         if(t == null) return ResponseEntity.notFound().build();
@@ -137,18 +137,18 @@ public ResponseEntity<?> handleDeleteJobById(@PathVariable String id) {
         if(!deleteRes) return ResponseEntity.notFound().build();
         return ResponseEntity.ok(Map.of("message", String.format("Job %s deleted!", id)));
     }
-    /* DEBUG:+NOTE:+TO-DO: ^ When I get to the stage where I start really expanding on the API endpoints (making this a deployable microservice),
+    DEBUG:+NOTE:+TO-DO: ^ When I get to the stage where I start really expanding on the API endpoints (making this a deployable microservice),
     I want to change the return value here slightly. In best practice, it's not supposed to be a 200 (OK) response, RESTful API
     design has it so that what I'd do here is return 204 (No Content) sign, which would imply "the resource was deleted successfully,
     there is no further content to return."
     DEBUG:+NOTE:+TO-DO: Re-scan over all the functions, honestly, and evaluate if my return codes are correct later. (Do some more reading into return codes, etc).
-    */
+
 
     // 6. This is for the [POST /api/clear]
     @PostMapping("/clear")
     public ResponseEntity<?> clearQueue() {
         queue.clear();
         return ResponseEntity.ok(Map.of("message", "All jobs in the queue cleared!"));
-    }
+    }*/
 
 }