feat(wazuh_decoder_rule_tool): guard against overfit prematches and free-text contamination - #106
Open
Hasitha9796 wants to merge 12 commits into
Open
feat(wazuh_decoder_rule_tool): guard against overfit prematches and free-text contamination#106Hasitha9796 wants to merge 12 commits into
Hasitha9796 wants to merge 12 commits into
Conversation
…ree-text contamination Add guardrails to the decoder generate/validate loop: - Phase-1 predecode awareness: postpredecode_remainder() and default_prematch_boundary() so parent prematches anchor on the post-strip remainder instead of the raw log. - detect_overfit_prematch(): reject prematches that hardcode a literal predecoded timestamp or month abbreviation with no \d/\w generalization (invisible to logtest against the training sample). - _xml_wellformed_error() and _extract_logtest_errors() to fail fast on malformed XML and surface logtest errors into the correction prompt. Keep free-text rejection notes out of training/RAG data: - build_dataset.py: _looks_like_osregex() heuristic rejects prose human notes that are not actual OS_Regex patterns; drop the hardcoded app-name prematch. - rag_engine.py: skip source="rejection_corrected" records so they are never surfaced as "Retrieved Real Wazuh Decoder Example". - train.jsonl: remove the two synthetic prose records.
…hed blog post Expand Quick Start into a full 8-step install flow matching the blog: add Prerequisites, Python 3.9+ install, Ollama install, and a Clone step; open the UI at https://<NodeIP>:8443. Keep existing notes (certs gitignore, RAG background build) and the DashScope/OpenRouter pointer.
…match Decoder generation produced output that Wazuh accepts but never fires, and /health reported a stopped manager as healthy. Parent decoders: - Derive <prematch> from the log's own header up to its first distinctive token (product tag in delimited formats, program: in syslog), keeping tags like LOGV3/APPAUTH literal while generalizing instance digits and month names. Previously the prematch fell back to the app name, which is usually absent from the log. - Verify every generated prematch against the sample before shipping it, via a new OS_Regex -> Python translator. OS_Regex inverts PCRE: \. is any-char and a bare . is a literal dot. - Synthesize a parent decoder when the model emits only children. Children naming a parent that nothing defines can never be reached. - Rewrite parent="x" as <parent>x</parent>. Wazuh ignores the attribute form, silently orphaning the child. - Force the verified prematch into the parent, inserting it when the model omitted it. Models paraphrase the prematch they are given and the result stops matching. - Stop emitting an empty <parent> line when the parent name is unknown. Field extraction: - Terminate = values at |. In f:ts=...|f:host=...|d:proto=... the first key swallowed the whole line and no later field was extracted at all. - Gate the colon key:value scan behind "the = scan found nothing", and reject keys that cannot be field names. A colon is not a separator in a line that already uses =; scanning anyway invented fields named 14, accesslog, 2026-08-01T14 and https. - Pair regex to children by position when the model renames a field, instead of leaving its invented bare (\S+) in place. Manager health: - Probe wazuh-logtest end-to-end instead of stat-ing the binary. wazuh-logtest exits 0 even when wazuh-analysisd is down and leaves its socket file on disk, so both the exit code and the filesystem check report a stopped manager as healthy. Feed it a probe line and inspect output for the connection-failure signature. Empty stdin never opened the socket at all. Also un-ignore tests/ (the blanket test* rule silently swallowed new test files) and ignore .env. 63 tests pass.
…lues
A parent <prematch> must select the log FAMILY, but both producers were
emitting one event's data. For `MSG#88213|type=EDR.ALERT|ts=...` the tool
generated `^MSG\p\d+\ptype\pEDR.ALERT`, so an EDR.INFO line from the same
source matched no decoder at all. This was not the model inventing a bad
pattern: analyze_logs_impl derives the prematch and the AI prompt hands it
over as "You MUST use <prematch>...", so the model reproduced it faithfully.
derive_parent_prematch: the pipe branch always took >=2 segments, pulling in
the second segment's value, and _PRODUCT_TAG_RE classified `EDR.ALERT` as a
vendor tag (like LOGV3/APPAUTH) and kept it literal. The whitespace fallback
had the same failure via a different route, reaching into `[REQ ...]`.
prematch_osregex_from_current_logs was worse and previously unnoticed:
default_prematch_boundary ends in `" ".join(tokens[:6])`, which splits on
whitespace, so a space-free delimited log came back whole -- every value in
the prematch.
Both now route through _header_zone(), which bounds the header at the first
`key=` (keeping the key, dropping the value) or `[`-delimited body block.
`{` and `(` are deliberately not stops so a `{SVC:name}` header tag survives.
Also fixed, found while verifying the above:
* generalize_prefix_literal kept a literal month when a syslog priority
prefix pushed it off position 0 (`<134>Aug 1 ...`), pinning the decoder to
one month. Months are marked before escaping -- substituting afterwards
fails because `\p` puts a letter in front of the month.
* Syslog space-pads single-digit days, and each space escaped to its own
\s+, so a prematch built from "Aug 1" rejected "Dec 25". Collapse runs, as
_generalize_prematch_prefix already did.
* detect_overfit_prematch caught literal timestamps and month names but not
pinned field values -- it returned None on both reported decoders. It now
takes an optional sample_log and rejects a prematch carrying any value from
it, accounting for `=` arriving as `\p`. Its tag regex also missed
<prematch type="pcre2"> entirely.
Verified against wazuh-logtest 4.14.5: the derived envelope matches EDR.INFO
and NET.FLOW siblings and rejects a wrong envelope; the syslog form matches
Aug/Dec/Jan/Nov. Confirmed end to end against the running server, not just in
a fresh interpreter -- the first fix looked correct in isolation while the
live app kept serving the old pattern from a pre-fix process.
…lds too
_header_zone knew two field boundaries, `key=` and `[`. A log in the
`KEY(value)` style hit neither, so the first `=` it found was buried inside a
later field:
2026-08-01 14:43:00; PRIORITY(CRIT); COMPONENT(storage-node-3); ...
DETAIL(dev=/dev/sdb; ...)
The zone ran to that `dev=`, and the derived prematch pinned both PRIORITY and
COMPONENT values -- matching only CRIT events from storage-node-3:
^\d+\p\d+\p\d+\s+\d+\p\d+\p\d+\p\s+PRIORITY\pCRIT\p\p\s+COMPONENT\pstorage\pnode\p\d+\p\p
_HEADER_ZONE_RE now also stops at an identifier bound to `(`. A bare `{` or `(`
is still not a boundary, so a `{SVC:name}` header tag survives. Derived
prematch is now the timestamp shape plus the literal `PRIORITY(` key:
^\d+\p\d+\p\d+\s+\d+\p\d+\p\d+\p\s+PRIORITY\p
Also fixed, found while checking the second producer on this format:
default_prematch_boundary matched its "program:" marker with `:\s*`, and the
zero-width case let the clock colon in "14:43:" satisfy it, cutting the header
mid-timestamp. Requires `:\s+` now, as derive_parent_prematch already did.
Both producers agree on this format again.
Verified against wazuh-logtest 4.14.5 (Phase 1 strips nothing here, so the
prematch stays `^`-anchored): WARN/INFO/DEBUG severities and other components
match, `SEVERITY(` and unrelated formats do not. Confirmed against the running
server, not just a fresh interpreter.
…d captures
Two separate defects on comma-delimited `key:value` logs, e.g.
DEV:TH-SENSOR-0442,SEQ:88213,T:...,temp:23.4C,evt:THRESHOLD_BREACH,crc:0x8A3F
1. The parent prematch pinned the entire record. _header_zone knew `key=`,
`KEY(` and `[`; this format has none of them, so the zone was the whole
line and the device id, event name, firmware and crc all ended up in the
prematch -- it matched exactly one reading from one sensor.
A colon cannot simply be added as a boundary: every clock has two, and
"14:49:" would cut the header mid-timestamp. Three guards make it safe --
the key must start with a letter (so a timestamp tail never qualifies),
at least three such pairs must be present (so a lone `{SVC:name}` tag is
untouched), and a colon introducing another key rather than a value does
not count, which keeps `LOGV3|f:ts=` reaching `ts=` instead of stopping at
the `f:` namespace.
Verified byte-identical output afterwards for the pipe, bracketed-body,
syslog and KEY(value) formats.
2. Child captures swallowed the rest of the record. build_split_regexes_from_
fields defaults to `(\S+)`, and a comma is non-space, so every field
captured everything after it:
\.+temp:(\S+) -> '23.4C,hum:61%,batt:3.71V,...,crc:0x8A3F'
OS_Regex backtracks when a literal follows the group -- confirmed against
wazuh-logtest -- so appending the record delimiter bounds the capture with
no move to pcre2. The final field of a record has no delimiter after it and
correctly gets none appended.
\.+temp:(\S+), -> '23.4C'
Validated end to end against wazuh-logtest 4.14.5: all nine fields decode from
the sample and from a different device/event, and the prematch still matches
both. Confirmed against the running server.
Measured the tool end to end against wazuh-logtest over 26 log formats, grading each run by whether the *generated* decoder fired rather than whether any decoder did. Five defects let broken decoders report success: * OS_Regex \p was modelled as full punctuation, but wazuh-logtest excludes `~ @ ^ _ / \` and a backtick. A `~PAYGW~` or `~AUDIT~` header generalized to \p verified clean via osregex_matches() and matched nothing in production. Restrict the class to what Wazuh implements and keep such characters literal when generalizing. * A `[` at position 0 collapsed the header zone to "", so derive_parent_prematch() returned None and the parent shipped with no prematch at all — every log opening with a bracket (epoch stamps, apache dates) lost one. * Weekday names stayed literal while months were generalized, pinning a decoder to Mondays. Neither the generalizer nor detect_overfit_prematch covered them, so it passed logtest against its own sample. * /api/ai/generate-validated computed needs_custom_decoder and ignored it, emitting a redundant decoder for logs Wazuh already decodes (json, sshd, fortigate, ...). The built-in wins Phase 2, so the generated decoder never fired while validation still reported success. Route those to a rule keyed with <decoded_as>, validated by asserting the generated rule id fires — the decoder validator cannot check that, since the built-in satisfies it regardless of what the rule does. * parse_logtest_output() searched the whole output for the first `id:`, picking up a Phase 2 decoded field instead of the Phase 3 rule. `id` is a documented Wazuh static field, so any decoder extracting an event code misreported rule_id. Scope the rule lookups to the Phase 3 block. Also carries the order-name normalization and _affix_match tightening that were already in the working tree.
Hasitha9796
force-pushed
the
hasitha/decoder-overfit-guardrails
branch
from
August 3, 2026 09:05
d7544d3 to
e535f95
Compare
…starts Two defects that stopped generated parents from firing on ISO8601-stamped logs. Found by running 20 `<ISO8601> <TAG> key=value ...` samples through the tool and checking with wazuh-logtest whether the generated decoder fired, rather than whether any decoder did: 0 of 20 fired before, 20 of 20 after. * build_candidate discarded the prematch analyze_logs_impl had already derived from the post-pre-decoding remainder and re-derived one from the raw log, which still carries the header Wazuh strips. The result anchored on the timestamp itself (`^\d+\p\d+\p\d+T\d+\p...`), so no log with a recognised timestamp could ever match. Prefer the verified prematch, falling back only when it does not apply. * postpredecode_remainder trusted wazuh-logtest's absent `hostname:` line. On the ISO8601 path logtest prints no hostname, yet Wazuh still consumes the token after the timestamp as one — a child decoder given `^(\S+)` captured `event=authentication`, not the `VPNGW01` tag before it. Every prematch was therefore anchored a token too early. Consume that token, but only for a cleanly recognised stamp: when the pre-decoder grabs a fixed 31 characters and slices the following field, the reported timestamp carries the debris and no token boundary can be trusted. Wazuh facts pinned down by these tests, both surprising enough to be worth recording: an ISO8601 stamp ending in `Z` is mis-parsed (fixed 31 characters, cutting into the next field) while `+00:00` parses cleanly at any length, and the token after the timestamp is consumed as a hostname that logtest never reports.
…amples
The RAG store indexed decoder metadata only — `decoder:… regex:… order:…` —
while retrieval queries with a raw log line. That compared a log against
OS_Regex syntax, so official decoders scored barely above unrelated feedback
rows, and every official doc carried an empty log_example. Without a sample,
a retrieved decoder can teach the model XML *style* but not the log -> regex
mapping, which is the part that matters.
ruleset/testing/tests/*.ini in the Wazuh repo already holds the samples the
project itself uses as ground truth (1986 of them, with expected decoder and
rule). They were never present locally because the repo cache pins its sparse
checkout to ruleset/decoders.
scripts/harvest_log_samples.py fetches that path, pushes every sample through
wazuh-logtest in batches, and keeps only pairs logtest confirms — recording the
decoder that actually fired and the fields it actually extracted. 1616 of 1986
verify. Two traps worth naming: JSON logs emit no `full event:` line at all, so
results are aligned positionally rather than keyed on the echoed event (keying
lost 504 samples), and a repeated `log 1 pass` key collapses into one
newline-joined value, so each line is split back out (another ~300).
The sample now leads each document's embedding text and populates log_example:
1089/1330 official docs (81%) carry a verified sample, 1472/1713 overall.
Measured with scripts/eval_rag_retrieval.py on held-out samples — any sample
indexed as a doc's log_example is excluded, since querying with a string that
is verbatim in the store measures memorisation, not retrieval. On logs that
need a real text decoder (n=578; builtin-json logs are excluded because no XML
decoder is the right answer for them):
metadata only (before) p@1 48.8% recall@3 55.2%
+ verified log example p@1 58.7% recall@3 60.0%
+ dedup (production path) p@1 66.6% recall@3 69.6%
Also fixed two latent bugs this surfaced:
* retrieve() could take down a request. `fields` metadata was truncated with
json.dumps(...)[:500], which cuts mid-element and leaves `["a", "bc` for
json.loads to raise on. Now truncated by dropping whole elements, with a
defensive parse for stores already written that way.
* /api/rag/status reported ready=false/count=0 in any worker that had not yet
served a retrieval, and kept reporting it after an out-of-process rebuild
invalidated the cached handle. It now lazy-attaches like retrieve() does.
Sibling decoders share a log sample, so top_k=3 returned the same log three
times — paying for three examples and teaching one. retrieve() over-fetches and
keeps the best-scoring doc per distinct sample.
…ds, verified
ml_order was routed through select_requested_fields(), which intersects it
against what the local extractor already found — so a retrieved decoder's
<order> could only ever reorder fields, never contribute one. The field most
worth having was exactly the one dropped: for an sshd-shaped failed login the
retrieved decoder says srcuser,srcip and the extractor finds only srcip.
Output was byte-identical whether the suggestion was correct, absent, or
nonsense:
ml_order=None -> order=['srcip']
ml_order=['srcuser','srcip'] -> order=['srcip']
ml_order=['totally','made','up'] -> order=['srcip']
propose_ml_order_fields() locates a value for each retrieved field name by its
label, and a proposal survives only if the regex the generator would actually
emit captures, in *every* sample log, exactly the value located in that log. A
pattern that fits only the first sample is overfitting — the failure mode this
path has to avoid — so it is rejected.
Proposals join the pool via common_fields, deliberately NOT requested_fields:
adding a name there flips selection to "requested only" and drops every
heuristic fallback field, so proposing srcuser would have cost us srcip.
osregex_captures() is new because osregex_matches() escapes parens and so
reports no match for any pattern with a capture group — it could never have
verified what a <regex> extracts. osregex_to_python() gains keep_groups, off by
default so prematch verification keeps its existing meaning.
Verified against real Wazuh (temporary install, logtest, removed):
Phase 2: name: 'mlfixprobe' srcip: '192.168.1.50' srcuser: 'admin'
Phase 2: name: 'mlfixprobe' srcip: '10.20.30.40' srcuser: 'carol'
Measured by scripts/eval_ml_order_proposals.py against logtest's own Phase 2
output for 1546 verified samples: 131 fields recovered, 57 additions the
official decoder did not have, 79 samples improved (5.1%). The remaining
additions are dominated by name normalisation where the captured value is
correct — checkpoint dstip where the official decoder said dst, cisco user
where it said username — and that 57 overcounts, since srcuser scores as an
addition when the official decoder emitted user for the same token.
Three guards kill the genuinely-wrong class, each pinned by a test:
* bare-space matching only for labels where it is the convention. "Unescaped
URL path matches" yielded url="path"; "dst outside:116.6.127.120" yielded a
dstip still carrying its interface prefix.
* quoted values honoured after `:` as well as `=`, so action:"Key Install" is
not truncated to "Key".
* structural words (from, for, invalid, user…) rejected for space matches
only. `Failed password for user from 172.18.1.1` names no user at all and
was offering "from" as the srcuser; `status=unknown` stays a valid value.
Also, the fuzzy affix fallback was relabelling values when fed retrieved names:
`timezone` matched `time`, `dstname` matched `dst`, `srcmac` matched `src`, each
emitting a decoder that captures a real value under a field name the log never
supported. select_requested_fields() gains allow_affix, off for the ML paths and
on by default so a human typing `ip` still finds `srcip`. Genuine synonyms that
had been working only through that fallback (proto/protocol, act/action,
username/user) are now explicit in FIELD_ALIASES. score_ml_decoder_template()
scores strictly too — it was crediting templates for fields selection would
then refuse, so templates ranked on matches they never had.
Costs 1.31ms on a worst-case 21-name order, against the ~950ms the ML model
already spends per analyze.
Hasitha9796
force-pushed
the
hasitha/decoder-overfit-guardrails
branch
from
August 3, 2026 19:59
fdded59 to
b448424
Compare
Grading a generation on `decoder_name != "unknown"` passed decoders that did nothing. A WAF child captured five groups against three <order> names, so Wazuh assigned none of them and the decoder extracted no fields; other samples were answered by a *built-in* decoder while the generated one never fired. Both shipped as validated. Validation now requires the generated decoder to be the one that fired, to extract fields when it declares an <order>, and to trigger its own rule. Two checks run before the install+logtest round trip, since logtest cannot see either: an <order>/capture-group arity mismatch, and an <order> name lifted out of event data (`A8` from an AP's MAC). Alongside that: - derive the parent prematch from every sample, not just the first. Two logs from one Aruba controller, cli[6005] and stm[6041], yielded a prematch pinned to `cli` that failed the sample supplied with it. - sanitize two OS_Regex constructs the model keeps emitting: `<` and `>` in pattern content (which opened an XML tag and made the decoder unparseable) become `\p`, and PCRE lazy quantifiers lose the `?` that OS_Regex reads as a literal. - point <decoded_as> at the parent decoder; keyed to a child it never fires, because logtest reports the parent. - catch three more overfit prematches: an opaque per-event id in a positional format, a 3+ digit literal (`^E0803` pinned a klog to August 3rd), and one carrying six or more literal words, which has run past the envelope into one event's body. Pinned-value detection also reads JSON `"key":"value"` and `key:value` records, not just `key=value` — a JSON prematch embedding ERROR, payments-api and jdoe had been passing clean.
The app has no decoder engine of its own — it drives the real wazuh-logtest binary — so without a running manager every validation is silently skipped. Install and start it before anything else, and install only the wazuh-manager package: the all-in-one installer deploys an indexer, dashboard and filebeat this tool never touches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Summary
Adds guardrails to the decoder generate/validate loop, keeps unverified free-text out of training/RAG data, and aligns the README install steps with the published blog post.
Overfit / Phase-1 predecode guardrails (
app/main.py)postpredecode_remainder()/default_prematch_boundary()— anchor parent prematches on the post-strip remainder (after Wazuh Phase-1 removes timestamp/hostname), not the raw log.detect_overfit_prematch()— reject prematches that hardcode a literal predecoded timestamp or month abbreviation with no\d/\wgeneralization. These passwazuh-logtestagainst the training sample yet fail on real traffic._xml_wellformed_error()/_extract_logtest_errors()— fail fast on malformed XML and surface logtest errors into the correction prompt.Keep free-text rejection notes out of training/RAG data
scripts/build_dataset.py—_looks_like_osregex()rejects prose human notes that aren't real OS_Regex patterns; drop the hardcoded app-name prematch.app/rag_engine.py— skipsource="rejection_corrected"records so they're never surfaced as a "Retrieved Real Wazuh Decoder Example".data/datasets/train.jsonl— remove the two synthetic prose records.Docs
https://<NodeIP>:8443).Test plan
wazuh-logtestvalidation loop via/api/ai/generate-validated.