perf(codex): skip out-of-range date directories when --since/--until is set#877
perf(codex): skip out-of-range date directories when --since/--until is set#877jleechan2015 wants to merge 4 commits intoryoppippi:mainfrom
Conversation
📝 WalkthroughWalkthroughAdds optional since/until date-range filtering across token usage loaders: commands now forward filters, codex data-loader gains directory-aware listing that prunes YYYY/MM/DD subtrees, and ccusage introduces mtime-based pruning and a DateFilter type. Changes
Sequence Diagram(s)mermaid Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR improves Codex token-usage loading performance by avoiding a full recursive **/*.jsonl glob when --since/--until is provided, instead pruning YYYY/MM/DD directory subtrees outside the requested range.
Changes:
- Extend
loadTokenUsageEvents()options to acceptsince/until. - Add
listSessionFiles()to enumerate only in-range date directories when filters are set (fallback to full glob when not set). - Thread
since/untilfromdaily,monthly, andsessioncommands intoloadTokenUsageEvents().
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| apps/codex/src/data-loader.ts | Adds date-range-aware session file listing and threads since/until into file discovery. |
| apps/codex/src/commands/daily.ts | Passes since/until into loadTokenUsageEvents() so pruning is applied. |
| apps/codex/src/commands/monthly.ts | Passes since/until into loadTokenUsageEvents() so pruning is applied. |
| apps/codex/src/commands/session.ts | Passes since/until into loadTokenUsageEvents() so pruning is applied. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| for (const year of (await tryReaddir(sessionsDir)).filter((e) => /^\d{4}$/.test(e))) { | ||
| if (sinceKey != null && `${year}1231` < sinceKey) { | ||
| continue; | ||
| } | ||
| if (untilKey != null && `${year}0101` > untilKey) { |
There was a problem hiding this comment.
When since/until is set, this implementation only traverses YYYY/MM/DD subdirectories and will ignore any .jsonl files stored directly under sessionsDir (or under non-date subdirs). The existing vitest fixtures in this file create sessions/project-1.jsonl and sessions/legacy.jsonl at the root, so running loadTokenUsageEvents({ since, until }) against a flat layout would incorrectly return no events. Consider also including glob('*.jsonl', { cwd: sessionsDir, absolute: true }) (and/or a small non-recursive fallback) alongside the pruned traversal so date filters don’t break non-date session layouts.
| async function listSessionFiles( | ||
| sessionsDir: string, | ||
| since: string | undefined, | ||
| until: string | undefined, | ||
| ): Promise<string[]> { | ||
| if (since == null && until == null) { | ||
| return glob(SESSION_GLOB, { cwd: sessionsDir, absolute: true }); | ||
| } | ||
|
|
||
| const sinceKey = since?.replaceAll('-', ''); | ||
| const untilKey = until?.replaceAll('-', ''); | ||
|
|
There was a problem hiding this comment.
listSessionFiles() introduces new range-pruning behavior but there are no tests exercising it (e.g., that it returns only files within [since, until], and that it still includes root-level .jsonl files if they exist). Since this file already contains vitest coverage for loadTokenUsageEvents, adding a focused test case that sets since/until and uses a dated YYYY/MM/DD fixture directory structure would help prevent regressions in the pruning logic.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@apps/codex/src/data-loader.ts`:
- Around line 192-249: The current listSessionFiles function ignores flat
session files in sessionsDir when since/until are set; restore compatibility by
explicitly globbing SESSION_GLOB at the root even in the filtered branch and
applying the same date filtering logic: inside listSessionFiles (before or
alongside the year/month/day traversal) call glob(SESSION_GLOB, { cwd:
sessionsDir, absolute: true }) to get rootFiles, derive a date key from each
root file's basename (e.g., strip extension and non-digits to produce YYYYMMDD
or YYYYMMDD-like key), compare that dateKey against sinceKey/untilKey the same
way you do for dateKey in the day loop, and push matching files into files; use
existing helpers (tryReaddir) and constants (SESSION_GLOB) so behavior is
consistent with the nested traversal.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 23a97fbc-4726-4244-8a50-b8460f101395
📒 Files selected for processing (4)
apps/codex/src/commands/daily.tsapps/codex/src/commands/monthly.tsapps/codex/src/commands/session.tsapps/codex/src/data-loader.ts
…is set Previously, loadTokenUsageEvents() always globbed **/*.jsonl across the entire sessions directory tree, which reads all historical data regardless of the requested date range. On large installations (35 GB+, 12 000+ files) this caused the process to be OOM-killed by the OS before producing any output. Sessions are stored as YYYY/MM/DD/*.jsonl. When --since or --until is provided, enumerate the year/month/day directories and prune entire subtrees that fall outside the range instead of performing a full recursive glob. Fall back to the recursive glob when no date filters are specified to preserve existing behaviour. Thread the since/until values from each command (daily, monthly, session) into loadTokenUsageEvents() so the directory-level filter is applied. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
jleechan2015
force-pushed
the
fix/date-filtered-session-loading
branch
from
b1d4efd to
e53a085
Compare
Reviewers flagged that listSessionFiles() with since/until set only walks YYYY/MM/DD subdirectories, causing flat session layouts to return no events. Fix by globbing *.jsonl directly under sessionsDir before the year/month/day traversal so both layouts are always covered.
Reviewer (Copilot + CodeRabbit) flagged that listSessionFiles() with since/until active skipped flat *.jsonl files stored directly under sessionsDir. Added a test that mixes a root-level flat.jsonl with a dated 2026/03/05/dated.jsonl and asserts both are returned when since is set — confirming the rootFiles glob fix covers the legacy layout.
There was a problem hiding this comment.
🧹 Nitpick comments (1)
apps/codex/src/data-loader.ts (1)
214-245: UseResult.try()for theglob()failure path too.Lines 214 and 245 switch back to ad-hoc
.catch(() => []), so file discovery now handles errors differently fromreaddir,stat, and JSON parsing in the same module. Keepingglob()on the sameResultpath will make failures consistent and easier to reason about.Suggested refactor
+ const tryGlob = async (cwd: string): Promise<string[]> => { + const result = await Result.try({ + try: glob("*.jsonl", { cwd, absolute: true }), + catch: (error) => error, + }); + return Result.isFailure(result) ? [] : result.value; + }; + // Preserve support for legacy flat layouts: include any *.jsonl files // stored directly under sessionsDir (not inside YYYY/MM/DD subdirs). - const rootFiles = await glob('*.jsonl', { cwd: sessionsDir, absolute: true }).catch(() => []); + const rootFiles = await tryGlob(sessionsDir); const files: string[] = [...rootFiles]; @@ - const dayFiles = await glob('*.jsonl', { cwd: dayDir, absolute: true }).catch(() => []); + const dayFiles = await tryGlob(dayDir); files.push(...dayFiles);As per coding guidelines, "Prefer
@praha/byethrowResult type over traditional try-catch for functional error handling" and "Use Result.try() for wrapping operations that may throw (JSON parsing, etc.)".🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@apps/codex/src/data-loader.ts` around lines 214 - 245, The glob() calls that set rootFiles and dayFiles currently use ad-hoc .catch(() => []) while other IO uses Result.try via the Result type; replace those ad-hoc catches by wrapping the glob(...) invocations with Result.try(...) (using the same Result import used elsewhere), propagate or unwrap the Result in the same pattern as tryReaddir/stat/JSON parsing, and assign the resulting array only after handling the Result (so rootFiles and dayFiles follow the same Result-based error path as the rest of the module).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@apps/codex/src/data-loader.ts`:
- Around line 214-245: The glob() calls that set rootFiles and dayFiles
currently use ad-hoc .catch(() => []) while other IO uses Result.try via the
Result type; replace those ad-hoc catches by wrapping the glob(...) invocations
with Result.try(...) (using the same Result import used elsewhere), propagate or
unwrap the Result in the same pattern as tryReaddir/stat/JSON parsing, and
assign the resulting array only after handling the Result (so rootFiles and
dayFiles follow the same Result-based error path as the rest of the module).
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: a58e98c7-d160-4ffe-a047-920647227c50
📒 Files selected for processing (4)
apps/codex/src/commands/daily.tsapps/codex/src/commands/monthly.tsapps/codex/src/commands/session.tsapps/codex/src/data-loader.ts
🚧 Files skipped from review as they are similar to previous changes (3)
- apps/codex/src/commands/daily.ts
- apps/codex/src/commands/monthly.ts
- apps/codex/src/commands/session.ts
Claude stores sessions as flat UUIDs under projects/<hash>/<uuid>.jsonl with no date directory hierarchy, so the date-tree traversal used for Codex is not applicable. Instead, filter by filesystem mtime after globbing: any file last written more than one day before --since cannot contain entries in the requested range and is skipped before any content is read. The 24-hour buffer absorbs timezone differences between local clock and UTC. Only --since is used for mtime pruning; --until is intentionally ignored because a file updated after --until may still hold entries from before it. Measured on a 28 800-file / 6.7 GB store with --since 7 days ago: Before: ~90 s (all files read via sortFilesByTimestamp) After: ~25 s (10 000 files pass mtime; rest pruned at stat level)
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
apps/ccusage/src/data-loader.ts (1)
1398-1407: ReuseglobUsageFiles()here instead of reimplementing the glob/prune path.This loop now duplicates the new enumeration logic but skips the per-path error handling and parallelization already baked into
globUsageFiles(). Keeping a single code path will prevent session-block loading from drifting from the daily/session loaders.Suggested simplification
- // Collect files from all paths, pruning by mtime when since is set - const allFiles: string[] = []; - for (const claudePath of claudePaths) { - const claudeDir = path.join(claudePath, CLAUDE_PROJECTS_DIR_NAME); - const files = await glob([USAGE_DATA_GLOB_PATTERN], { - cwd: claudeDir, - absolute: true, - }); - const filtered = await filterFilesByMtime(files, options?.since); - allFiles.push(...filtered); - } + const allFiles = ( + await globUsageFiles(claudePaths, { + since: options?.since, + until: options?.until, + }) + ).map(({ file }) => file);🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@apps/ccusage/src/data-loader.ts` around lines 1398 - 1407, The loop that enumerates and prunes files duplicates logic already implemented in globUsageFiles; replace the manual per-claudePath glob + filterFilesByMtime logic (the block that builds allFiles from claudePaths and calls filterFilesByMtime) with a single call to globUsageFiles(...) to obtain the same pruned list (respecting options?.since), assign its result to allFiles (or merge as needed), and remove the duplicated code so you leverage globUsageFiles's per-path error handling and parallelization.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@apps/ccusage/src/data-loader.ts`:
- Around line 4746-4769: The tests use dynamic imports for node:fs/promises and
createFixture; replace those with the existing static imports: add utimes to the
top-of-file import that already imports from 'node:fs/promises' and remove the
three await import(...) calls (the ones that import { utimes } at line ~4747 and
the two that import createFixture at ~1580 and ~4778), then update references to
use the statically imported utimes and createFixture symbols directly (no await
import). Ensure globUsageFiles and related test logic remain unchanged.
- Around line 703-708: Top-level imports currently include readFile and stat
from 'node:fs/promises' but tests use await import('node:fs/promises') to get
utimes; update the static import to also import utimes (add utimes to the
existing import list that contains readFile and stat) and remove the dynamic
import usage—replace any await import('node:fs/promises') calls with direct
references to utimes from the top-level import (ensure code that used the
dynamic import now calls utimes() directly).
---
Nitpick comments:
In `@apps/ccusage/src/data-loader.ts`:
- Around line 1398-1407: The loop that enumerates and prunes files duplicates
logic already implemented in globUsageFiles; replace the manual per-claudePath
glob + filterFilesByMtime logic (the block that builds allFiles from claudePaths
and calls filterFilesByMtime) with a single call to globUsageFiles(...) to
obtain the same pruned list (respecting options?.since), assign its result to
allFiles (or merge as needed), and remove the duplicated code so you leverage
globUsageFiles's per-path error handling and parallelization.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: ef7eb3a5-f9f5-4999-86da-462b9af789d9
📒 Files selected for processing (1)
apps/ccusage/src/data-loader.ts
| function parseDateKey(yyyymmdd: string): Date { | ||
| const year = Number.parseInt(yyyymmdd.slice(0, 4), 10); | ||
| const month = Number.parseInt(yyyymmdd.slice(4, 6), 10) - 1; | ||
| const day = Number.parseInt(yyyymmdd.slice(6, 8), 10); | ||
| return new Date(year, month, day); | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -nP '\bctx\.values\.(since|until)\b|\b(since|until)\b.*replace(All)?\(|type DateFilter|filterByDateRange\(' apps/ccusage/src -g '*.ts'Repository: ryoppippi/ccusage
Length of output: 1146
🏁 Script executed:
#!/bin/bash
# Get the DateFilter type definition and LoadOptions around line 766
sed -n '766,800p' apps/ccusage/src/data-loader.ts
# Find where parseDateKey is called
rg -n 'parseDateKey' apps/ccusage/src -g '*.ts' -B 2 -A 2
# Check how since/until parameters are defined in commands
grep -n 'since\|until' apps/ccusage/src/commands/session.ts | head -20
grep -n 'since\|until' apps/ccusage/src/commands/blocks.ts | head -20Repository: ryoppippi/ccusage
Length of output: 2575
🏁 Script executed:
#!/bin/bash
# Look for command parameter definitions for since/until
grep -n -B 5 -A 5 'since.*until\|until.*since' apps/ccusage/src/commands/session.ts | head -40
grep -n -B 5 -A 5 'since.*until\|until.*since' apps/ccusage/src/commands/blocks.ts | head -40
# Check if there are any date parsing/normalization in command setup
grep -n 'parseDate\|Date\.parse\|date.*parse\|ISO\|format' apps/ccusage/src/commands/session.ts
grep -n 'parseDate\|Date\.parse\|date.*parse\|ISO\|format' apps/ccusage/src/commands/blocks.ts
# Look at the full context around since/until in session.ts
sed -n '60,85p' apps/ccusage/src/commands/session.tsRepository: ryoppippi/ccusage
Length of output: 3260
🏁 Script executed:
#!/bin/bash
# Find where the since/until parameters are defined in the command setup
grep -n -B 15 'ctx.values.since' apps/ccusage/src/commands/session.ts | grep -A 15 'option\|param\|since\|until' | head -50
# Look for parameter definitions in blocks.ts as well
grep -n -B 15 'ctx.values.since' apps/ccusage/src/commands/blocks.ts | grep -A 15 'option\|param\|since\|until' | head -50
# Try searching for where since/until are declared as command parameters
rg -n 'name.*since|name.*until|key.*since|key.*until' apps/ccusage/src/commands -g '*.ts' -B 2 -A 5Repository: ryoppippi/ccusage
Length of output: 123
🏁 Script executed:
#!/bin/bash
# Get the full session.ts file structure to see where params are defined
head -150 apps/ccusage/src/commands/session.ts | tail -100
# Look for create/createCommand patterns and options definitions
rg -n 'createCommand|option|param' apps/ccusage/src/commands/session.ts | head -20Repository: ryoppippi/ccusage
Length of output: 2900
🏁 Script executed:
#!/bin/bash
# Look at the beginning of session.ts to see imports and command setup
head -80 apps/ccusage/src/commands/session.ts
# Search for where options are defined - maybe they're from a shared utility
rg -n 'export.*option|export.*param' apps/ccusage/src/commands -g '*.ts'
# Look for any shared command options file
find apps/ccusage/src/commands -name '*option*' -o -name '*param*'Repository: ryoppippi/ccusage
Length of output: 2537
🏁 Script executed:
#!/bin/bash
# Find the _shared-args file
find apps/ccusage/src -name '*shared*args*' -o -name '_shared*'
# Look at the shared args file
cat apps/ccusage/src/_shared-args.ts
# Also check what sharedCommandConfig contains
rg -n 'sharedCommandConfig' apps/ccusage/src/_shared-args.ts -A 30Repository: ryoppippi/ccusage
Length of output: 3360
🏁 Script executed:
#!/bin/bash
# Check the filterDateSchema definition
grep -n 'filterDateSchema' apps/ccusage/src/_types.ts -B 5 -A 10
# Also look for the full valibot schema definition
rg -n 'export.*filterDateSchema|const.*filterDateSchema' apps/ccusage/src -g '*.ts' -A 10Repository: ryoppippi/ccusage
Length of output: 4901
🏁 Script executed:
#!/bin/bash
# Check all usages of parseDateKey to see if there are other callers
rg -n 'parseDateKey' apps/ccusage/src -g '*.ts' -B 3 -A 3
# Check the test at lines 4746-4769
sed -n '4745,4770p' apps/ccusage/src/data-loader.tsRepository: ryoppippi/ccusage
Length of output: 2174
🏁 Script executed:
#!/bin/bash
# Verify that parseDateKey is only called with validated input
# Search for all function calls to parseDateKey
rg -n 'parseDateKey\(' apps/ccusage/src -g '*.ts'
# Verify that since/until only come from the validated command-line parameters or DateFilter type
# Check if DateFilter type is used anywhere else
rg -n 'DateFilter\|since.*:.*string\|until.*:.*string' apps/ccusage/src -g '*.ts' -B 2 -A 2 | head -50
# Check if there's any other way since/until are set besides the command parsing
rg -n '\bsince\s*:\s*|\.since\s*=' apps/ccusage/src -g '*.ts' | head -20Repository: ryoppippi/ccusage
Length of output: 1329
🏁 Script executed:
#!/bin/bash
# Get full test code to see how it's structured
sed -n '4746,4770p' apps/ccusage/src/data-loader.tsRepository: ryoppippi/ccusage
Length of output: 1184
🏁 Script executed:
#!/bin/bash
# Check how fs imports are typically handled in this file
grep -n "import.*fs\|from.*node:fs" apps/ccusage/src/data-loader.ts | head -10
# Also check if fs is already imported elsewhere in the file
head -100 apps/ccusage/src/data-loader.ts | grep -n "import"Repository: ryoppippi/ccusage
Length of output: 1540
Remove dynamic import of node:fs/promises and add utimes to the top-level import.
Line 4747 uses await import('node:fs/promises') to get utimes, which violates the no-dynamic-imports constraint for apps/ccusage/**/*.ts. Since readFile and stat are already imported from node:fs/promises at line 16, add utimes to that import statement instead.
-import { readFile, stat } from 'node:fs/promises';
+import { readFile, stat, utimes } from 'node:fs/promises';Then replace the dynamic import in the test:
it('skips files whose mtime is more than one day before since', async () => {
- const { utimes } = await import('node:fs/promises');
-
await using fixture = await createFixture({🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@apps/ccusage/src/data-loader.ts` around lines 703 - 708, Top-level imports
currently include readFile and stat from 'node:fs/promises' but tests use await
import('node:fs/promises') to get utimes; update the static import to also
import utimes (add utimes to the existing import list that contains readFile and
stat) and remove the dynamic import usage—replace any await
import('node:fs/promises') calls with direct references to utimes from the
top-level import (ensure code that used the dynamic import now calls utimes()
directly).
| it('skips files whose mtime is more than one day before since', async () => { | ||
| const { utimes } = await import('node:fs/promises'); | ||
|
|
||
| await using fixture = await createFixture({ | ||
| 'base/projects/proj/old/usage.jsonl': 'old-data', | ||
| 'base/projects/proj/new/usage.jsonl': 'new-data', | ||
| }); | ||
|
|
||
| const oldFile = fixture.getPath('base/projects/proj/old/usage.jsonl'); | ||
| const newFile = fixture.getPath('base/projects/proj/new/usage.jsonl'); | ||
|
|
||
| // Set old file mtime to 5 days before since, new file mtime to 1 day before since. | ||
| const since = new Date('2026-03-05'); | ||
| const oldMtime = new Date(since.getTime() - 5 * 24 * 60 * 60 * 1000); | ||
| const newMtime = new Date(since.getTime() - 12 * 60 * 60 * 1000); // 12h before (within buffer) | ||
| await utimes(oldFile, oldMtime, oldMtime); | ||
| await utimes(newFile, newMtime, newMtime); | ||
|
|
||
| const results = await globUsageFiles([fixture.getPath('base')], { since: '20260305' }); | ||
|
|
||
| // Old file (5d before since) should be pruned; new file (12h before, within 1d buffer) kept. | ||
| expect(results).toHaveLength(1); | ||
| expect(results[0]!.file).toContain('new'); | ||
| }); |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -nP 'await\s+import\s*\(' apps/ccusage/src -g '*.ts'Repository: ryoppippi/ccusage
Length of output: 339
🏁 Script executed:
head -50 apps/ccusage/src/data-loader.tsRepository: ryoppippi/ccusage
Length of output: 1751
🏁 Script executed:
sed -n '1575,1585p' apps/ccusage/src/data-loader.tsRepository: ryoppippi/ccusage
Length of output: 295
🏁 Script executed:
sed -n '4773,4783p' apps/ccusage/src/data-loader.tsRepository: ryoppippi/ccusage
Length of output: 568
Replace all dynamic imports in this file with static imports.
This file uses await import() three times (lines 1580, 4747, and 4778), which violates the repo's policy in apps/ccusage/**/*.ts. Both node:fs/promises and fs-fixture are already statically imported at the top of the file, so these should be used instead:
Suggested fixes
Line 4747 — add utimes to the existing import:
-import { readFile, stat } from 'node:fs/promises';
+import { readFile, stat, utimes } from 'node:fs/promises';Lines 1580 and 4778 — remove the dynamic imports since createFixture is already statically imported:
describe('loadSessionUsageById', async () => {
- const { createFixture } = await import('fs-fixture'); describe('calculateContextTokens', async () => {
- const { createFixture } = await import('fs-fixture');🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@apps/ccusage/src/data-loader.ts` around lines 4746 - 4769, The tests use
dynamic imports for node:fs/promises and createFixture; replace those with the
existing static imports: add utimes to the top-of-file import that already
imports from 'node:fs/promises' and remove the three await import(...) calls
(the ones that import { utimes } at line ~4747 and the two that import
createFixture at ~1580 and ~4778), then update references to use the statically
imported utimes and createFixture symbols directly (no await import). Ensure
globUsageFiles and related test logic remain unchanged.
Bug
Root cause (Codex)
loadTokenUsageEvents()inapps/codex/src/data-loader.tsusedglob('**/*.jsonl', { cwd: sessionsDir })— a full recursive scan unconditionally, regardless of--since/--until. Thesince/untilvalues were forwarded only to the report-building step, not to the file-discovery step.Root cause (ccusage / Claude sessions)
globUsageFiles()inapps/ccusage/src/data-loader.tsglobbed all~/.claude/projects/**/*.jsonlfiles (28 800+ files / 6.7 GB on large installs) before any date filtering. Claude sessions have no date directory hierarchy, so tree-traversal pruning is not applicable.Observed failure
On large installs the session directory exceeds available memory; macOS OOM-kills the Node process:
Changes
Codex — directory-tree traversal pruning (
apps/codex/src/data-loader.ts)Added
listSessionFiles()which enumeratesYYYY/MM/DDdirectories and skips entire subtrees outside the requested date range before globbing. Falls back to**/*.jsonlwhen no date filter is given. Also preserves legacy flat*.jsonlfiles at the sessions root (reviewer feedback).Measured on 35 GB / 12 000+ file store with
--since 7d:ccusage — mtime-based file pruning (
apps/ccusage/src/data-loader.ts)Added
filterFilesByMtime()which usesstat()to skip files whose mtime is more than one day before--since. The 24-hour buffer absorbs timezone differences.--untilis intentionally not used for mtime pruning (a file updated after--untilmay still hold entries from before it).Applied in
globUsageFiles(),loadDailyUsageData(),loadSessionData(), andloadSessionBlockData().Measured on 28 800-file / 6.7 GB Claude store with
--since 7d:TDD methodology
RED → GREEN for each fix. In-source vitest tests added:
skips date directories outside the since/until range+includes root-level *.jsonl files when date filters are activeskips files whose mtime is more than one day before since(usesutimes()to set deterministic mtime on fixture files)Impact
ccusage-codex --since 7don 35 GB storeccusage --since 7don 6.7 GB storeSummary by CodeRabbit