V2 Consideration: Matching Arrays with sigma

[humpalum]

Matching Arrays with sigma

Problem

Sigma currently lacks the capability to match values within arrays in log events. This limitation restricts its effectiveness in matching complex log data that includes arrays. For example:

{
    "message": "Some Message",
    "process": "Some Process",
    "connections": [
        "123.1.1.1",
        "123.1.2.2",
        "123.3.3.3"
    ]
}

It is not possible to match a value in that array for example with:

connection: "123.1.1.1"

Proposed Solution

Enhance Sigma with key modifiers for array matching, such as |arrayAny, |arrayAll, |arrayOne, and |arrayNone. This would allow for more flexible and powerful matching scenarios.

connections|arrayAny: 123.1.1.1

This would return true if any element in the connections array matches "123.1.1.1".

connections|arrayAll|startswith: 123

This would return true if all elements in the array start with "123".

--> Inspired by https://expr-lang.org/docs/Language-Definition#array-functions

Further Enhancement

Even further we could allow matching of nested keys on arrays of objects:

{
    "message": "Some Message",
    "process": "Some Process",
    "connections": [
        {"protocol":"TCP","ip":"123.1.1.1"},
        {"protocol":"TCP","ip":"123.1.2.2"},
        {"protocol":"UDP","ip":"123.3.3.3"}
    ]
}

Allowing to "chain" multiple keys would allow to match on array of objects. Taking above example, we could write detections like this:

# Check if there is one UDP connection beginning with 123
connections|arrayAny:
    protocol: "UDP"
    ip|startswith: "123"
# --> Would return true

or

# Check if all connections beginning with 123 are using TCP
connections|arrayAll:
    protocol: "TCP"
    ip|startswith: "123"
# --> Would return false

I can see how this is a big change of current sigma philosophy and I am not sure how implementable this is in backends but taking elasticsearch as an example, it would work on nested objects: Nested Fields

[norbert791]

I would like to bump this topic and add a few things from myself.

Rationale

I believe we have a good reason to add array support to sigma. Many native log sources use arrays - including (but definitely not limited to):

• K8s audit logs
• Cloudtrail
• Azure Activity logs
• GCP Activity logs
• Okta

It's worth to note that only Azure would not benefit from the Further Enhancement feature (proposed by @humpalum) since it does not appear to use JSON objects inside arrays. Additionally, note that sigma does already have rules for most of the log sources from the list above.

Additional observations

I am not sure how sigma handles nested fields at the moment. We do have some rules that use . to access fields that are nested in original log sources.

Proposed solution

The proposed solution roughly aligns with what @humpalum suggests, but I propose different syntax (for the sake of discussion).

The format below is somewhat inspired by keywords search

Let us consider the example by @humpalum once more:

{
    "message": "Some Message",
    "process": "Some Process",
    "connections": [
        {"protocol":"TCP","ip":"123.1.1.1"},
        {"protocol":"TCP","ip":"123.1.2.2"},
        {"protocol":"UDP","ip":"123.3.3.3"}
    ]
}

A rule that matches events with such schema if it contains any TCP connection with IP address "123.1.1.1" would look like this:

selection:
  matchNode:
    connections[any].protocol: "TCP"
    connections[any].ip: "123.1.1.1"

which should be interpreted as a 'flattened' version of @humpalum's syntax. Consequently, the rule below

selection:
  matchNode:
    connections[any].protocol: "TCP"
    connections[all].ip: "123.1.1.1"

would mean **match events where any connection is a TCP, but all of them have IP "123.1.1.1".

This syntax can be nested easily. Let's consider another example:

{
   "configName": "Some Name",
   "rules": [
       {"type":"allow", "ip":["123.1.1.1", "123.1.1.2", "123.1.1.3"]},
       {"type":"allow", "ip":["123.1.2.2"]},
       {"type":"block","ip":["124.3.3.3"]}
   ]
}

And a rule

selection:
  matchNode:
    rules[any].type: "allow"
    rules[any].ip[all]|startswith: "123.1.1"
    rules[all].ip[all]|startswith: "123"

This time, the rule matches any events where all rule ips start with "123". Additionally there must be at least one rule object R which satisfies:

• R.type == "allow"
• For each s in R.ip array: s starts with "123.1.1"

Pros and Cons

I believe this syntax might be a bit more consistent with current sigma format. It would also not increase the depth of rules much. It also allows fine grained control over how nested logs should be matched. On the flip side it might be less straightforward and more verbose than @humpalum's solution.

Regardless of the format, I would really embrace any official support for nested formats.

[mostafa]

Following up here (and on a side conversation with @thomaspatzke on Discord): I've written a full SEP that synthesizes the proposals in this thread, SEP: Array Matching in Sigma #212.

The core idea is to treat "array matching" as two separate problems:

1. Quantification: must any or all members satisfy a predicate (the common, flat-array case).
2. Element correlation: must the same element satisfy several predicates (the hard, array-of-objects case).

The proposal:

• Makes any-member matching the default for plain field expressions, with no new syntax. This is already how Elasticsearch/Splunk/KQL behave, and it's @thomaspatzke's "any is the default."
• Adds one explicit construct, an object-scope quantifier block (field[any]: / field[all]: with a nested map) for correlation. This is @humpalum's nested-block idea, with the quantifier moved onto the path (bracket notation from @norbert791) so it never collides with the existing all value-list modifier.
• Intentionally declines shared-prefix correlation across separate keys (e.g. connections[any].protocol and connections[any].ip as two keys), because nothing binds them to the same element and it doesn't convert portably. Correlation goes through the block instead.

The deciding factor was convertibility: the block lowers to a single canonical primitive per backend (Elastic nested, KQL mv-apply, SQL jsonb_array_elements + EXISTS, Splunk mvexpand), while the shared-prefix form lowers to none of them cleanly, which is exactly why engines diverge today.

Open questions I'd value feedback on: empty/missing semantics for [all], [any]/[all] brackets vs *, negation push-down, and whether one/none are worth including.

I'm building a reference implementation in rsigma (evaluator + converter) to validate the semantics across backends and feed findings back here.

[tg]

Just wanted to mention that we ran into the same problem at AlphaSOC and ended up introducing "recursive array matching": https://docs.alphasoc.com/detections_and_findings/sigma/supported_features/arrays/

One of our main concerns was preserving bindings between fields belonging to the same array element. It’s great to see that @mostafa's proposal explicitly addresses this.

One aspect we found valuable in our approach is that array elements are evaluated by recursively applying the Sigma detection model to each element. In other words, array elements behave like "mini-events" and existing Sigma concepts can be reused inside arrays. This allows for the same rule complexity as the base object, including negations and more complex conditions on specific fields. Apart from introducing a single arrayAll modifier, we largely achieved this by reusing existing Sigma semantics rather than creating a separate array matching model.

For example, assuming log entry with array of connections, “is there a non-TCP connection in 123.1.0.0/16?” can be expressed as:

detection:
  condition: selection
  selection:
    connections:
      condition: selection and not filter
      selection:
        ip|cidr: "123.1.0.0/16"
      filter:
        protocol: "TCP"

We could imagine adding syntax sugar on top of this in the future (potentially converging towards something similar to other proposals), but we wanted to establish a fully expressive foundation first. One thing we’re currently missing is support for matching specific indices. So far we haven’t seen detections where a particular array position has special meaning, but it’s certainly an interesting use case and worth considering.

Sharing this as another approach to the problem that exists in the wild. I’m glad Sigma is getting closer to supporting arrays, as this is clearly something users need.

[mostafa]

@tg This is great, and the recursive "element as a mini-event" framing is the most expressive foundation in the thread. It also cleanly subsumes the block in my SEP #212: field[any]: {map} is just your recursive body with an implicit condition: that ANDs every item.

The capability my block is missing is exactly the one yours adds: per-element boolean logic, especially negation. My block is conjunction-only, so "is there a connection in 123.1.0.0/16 that is not TCP" has no form. The two-key workaround is wrong because it drops the binding:

# WRONG: two independent scopes, not "the same element is in-CIDR and non-TCP"
selection:
  connections[none]:
    protocol: "TCP"
  connections[any]:
    ip|cidr: "123.1.0.0/16"

Your recursive body expresses it directly:

selection:
  connections[any]:
    condition: in_cidr and not is_tcp
    in_cidr:
      ip|cidr: "123.1.0.0/16"
    is_tcp:
      protocol: "TCP"

And it still lowers to the same canonical primitive, just with a richer inner predicate: EXISTS (SELECT 1 FROM jsonb_array_elements(connections) e WHERE (e->>'ip')::inet <<= '123.1.0.0/16' AND NOT (e->>'protocol' = 'TCP')), Elastic nested, KQL mv-apply. The array primitive is unchanged, so convertibility does not argue against it.

So I think the two proposals converge rather than compete:

• Recursive body (yours): the expressive core, a nested detection per element with full and/or/not.
• Bracket selectors (mine): ergonomic sugar for the common cases, plus the one thing you flagged as missing, positional indexing (args[0], args[-1]).

Concretely: the block body is "a nested Sigma detection" in full generality; the [any]/[all]/[all_or_empty]/[none] key stays as the quantifier, and the body may be either the shorthand map (implicit-AND, the 90% case) or a full condition: + named-selections block (your model, for negation/disjunction).

@thomaspatzke does this scoping match what you'd want for the spec? My one worry is grammar surface: a recursive condition: inside a detection item is a bigger change than the conjunction map, and we'd need to pin element-scoped selection names and how nested quantifiers compose with an inner condition. If you'd rather keep v1 to the conjunction map and track per-element boolean logic as a follow-on, that works too. I can validate both shapes in the rsigma reference implementation and report back which lowers cleanly across backends.

[thomaspatzke]

Both approaches have their advantages and disadvantages, mainly simplicity vs power. It wouldn't be the first time where Sigma supports two concurrent approaches with different capabilities. In Sigma correlations there are basic and extended conditions and both co-exist. So I'm totally fine with introducing both of them. Especially the one from the SEP should be relatively easy to implement, at least for the conversion use case. Finally, the experiences from basic correlation conditions helped to implement the extended ones.

[mostafa]

I'll update the proposal to include the dual approach.

[mostafa]

Done.

[mostafa]

Quick update on the array-matching SEP (#212) as suggested by @frack113: I added a Worked Examples section that pairs real-world log entries with rules that match them, drawn from AWS CloudTrail, Kubernetes audit, Okta system log, Elastic ECS, and network telemetry. Between these and the Problem Statement logs they cover every construct: implicit any-member, the basic and extended object-scope blocks, all four quantifiers, and positional indexing (including negative).

To keep logs and rules traceable, each example log is labeled case_N and every matching rule carries a # log: case_N comment as its first line (just a YAML comment, not a new key), so the pairs double as conformance vectors that the rsigma and pySigma implementations can test against directly. Feedback welcome, especially on whether the example log shapes match what you see in the wild.

[mostafa]

(The SEP is being constantly updated to reflect all perspectives and use-cases. Therefore, let's continue talking on the SEP issue itself, so that the history is preserved in a single place.)