Skip to content

feat: detect log pattern distribution changes over time #19

@STRRL

Description

@STRRL

Summary

Given a long-duration log file, after pattern matching, analyze how the distribution of log patterns changes over time and pinpoint the moments when shifts occur.

Motivation

In real-world incident investigation, the most valuable signal is often not "what patterns exist" but "when did the pattern mix change." For example:

  • At 14:30, connection-timeout pattern suddenly spikes from 2/min to 200/min
  • At 14:28, a new pattern oom-killer-invoked appears for the first time
  • At 14:35, health-check-ok pattern disappears entirely

These transition points are exactly where engineers should focus. Currently lapp gives a static summary of all patterns, but doesn't show temporal dynamics.

Proposed Behavior

lapp workspace add-log ./investigation production.log
# After pattern matching, lapp automatically:
# 1. Buckets matched lines into time windows
# 2. Computes per-pattern frequency in each window
# 3. Detects significant changes (new pattern, disappeared pattern, frequency spike/drop)
# 4. Outputs a timeline of change points

Output (in workspace)

Could be a notes/timeline.md or patterns/<id>/timeline.md per pattern:

## Pattern Distribution Changes

### 14:28:03 — New pattern appeared
- `oom-killer-invoked` first seen (0 → 12/min)

### 14:30:15 — Frequency spike
- `connection-timeout` jumped from 2/min to 200/min (+9900%)

### 14:35:00 — Pattern disappeared
- `health-check-ok` dropped from 30/min to 0

Design Considerations

  • Time extraction: Need to parse timestamps from log lines (or use line number as proxy when timestamps are unavailable)
  • Window size: Auto-detect reasonable bucket size based on log duration, or let user configure
  • Change detection: Could be simple (threshold-based) or statistical (e.g. z-score against rolling baseline)
  • Relationship to feat: AI-friendly workspace for Claude Code / Codex #17: Output should fit into the workspace structure — probably a file in notes/ that agents can read and investigate further

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions