feat(state): add incremental state persistence via --state flag#1
Open
PseudoSky wants to merge 1 commit into
Open
feat(state): add incremental state persistence via --state flag#1PseudoSky wants to merge 1 commit into
PseudoSky wants to merge 1 commit into
Conversation
Adds checkpoint-based incremental processing so repeated invocations on a growing log file only parse new lines (O(n_new) instead of O(n_total)). New CLI flags: --state <FILE> Load/save analysis checkpoint (byte offset + Drain model) --no-update-state Load checkpoint read-only without advancing it --state-max-age <DUR> Evict clusters not seen within duration at save time (e.g. 7d, 24h) --max-patterns (existing) now also caps the state file cluster count when --state is active. State file (JSON) persists: byte offset for seek-based incremental reads, Drain cluster templates with first/last_seen timestamps, param_baselines, temporal_baselines, and drain_config including max_patterns so subsequent runs honour the saved ceiling. On load, saved templates are replayed into a fresh DrainTree to preserve cluster continuity across runs. Anomaly detection uses loaded baselines: known templates flag new patterns; historical_rates (events/hr per template) trigger rate-spike anomalies when current rate exceeds 3x the baseline. Handles: log rotation (file shorter than offset → fresh run), version mismatch (ignores stale checkpoint), stdin mode (offset tracking disabled with info message), atomic saves (tmp file + rename to prevent corruption on mid-write kill). New module: src/state.rs — StateFile, PersistedCluster, DrainConfig, TemporalBaseline, parse_duration(), evict_stale(), cap_clusters(), save_atomic(). Tests added for rate-spike detection and DrainTree template continuity.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Added to enable incremental log analysis across repeated invocations. Each run processes only new content and updates a persistent state file with cluster templates and temporal baselines, so anomaly detection improves over time without reprocessing old data.
Example: passive shell command monitoring via zsh
preexecAdd to
~/.zshrc:preexecfires before every command with the command string as$3. Each invocation sends one line to logoscope; the state file accumulates pattern history so the first time an unusual command class appears it surfaces as anew_patternanomaly.Summary
--state <file>flag for incremental log processing — on each run, logoscope seeks to the saved byte offset (file mode) or loads historical baselines (stdin mode) and processes only new contentinsert_maskedreplay (drain_rs v0.3 is not serde-compatible)--stateis loaded, current events/hr per template is compared against historical baseline; ≥3× ratio emits arate_spiketemporal anomalyNew flags
--state <file>--no-update-state--state--state-max-age <dur>--state7d,24h)Files changed
logoscope/src/state.rs— new module:StateFile,PersistedCluster,DrainConfig,TemporalBaseline,parse_duration, atomic save, eviction/cap logiclogoscope/src/ai.rs—SummarizeOptsgainshistorical_rates; newsummarize_lines_with_preseeded_drain; rate-spike detection in temporal analysislogoscope/src/bin/logoscope.rs—--stateCLI wiring, byte-offset seek with inode rotation detection, seed DrainAdapter construction, state merge/savelogoscope/src/drain_adapter.rs— exposecluster_count()logoscope/src/lib.rs— addpub mod statelogoscope/tests/ai_anomalies_output_tests.rs— rate-spike detection testslogoscope/tests/drain_adapter_tests.rs— template continuity test for pre-seeded drainlogoscope/tests/state_rotation_tests.rs— integration tests for inode-based rotation detectionTest plan
cargo testpasses (all new tests green)--state /tmp/s.jsoncreates the file; second run on same file seeks to offset and skips old lines--state-max-age 1sevicts all clusters on a file not touched in >1s--no-update-stateleaves state file unchanged after a run