fix: Splunk Add-On merges distinct incidents due to UUID collision with imprecise event timestamps

[fruitcakej]

We have identified a critical issue with the OpenCTI Splunk Add-On where it incorrectly updates existing OpenCTI incidents instead of creating new ones for distinct Splunk security events. This occurs when multiple Splunk events trigger the same use case at the exact same second, leading to a collision in the generated STIX Incident ID.

This behavior results in data loss, corruption, and incorrect incident tracking, especially prevalent in high-volume environments like MSSPs where multiple similar alerts can fire concurrently.

Affected Components & Versions

• OpenCTI Splunk Add-On: Version 1.1.6 (and likely earlier/later versions using the same ID generation logic)
• OpenCTI Platform: Version 6.9.28 (behavior is likely platform-agnostic regarding STIX ID updates)

Steps to Reproduce

1. Configure Splunk Add-On: Ensure the Splunk Add-On for OpenCTI is configured to create incidents based on Splunk alerts.
2. Trigger Use Case (Event A): A Splunk use case is triggered for Customer A at a specific time (e.g., _time=1771487340 (epoch without milliseconds), name="Malicious IP Observed").
3. OpenCTI Incident Created: A new case/incident is created in OpenCTI for Customer A. (Optionally, a custom connector might trigger to add a short ID to the title).
4. Trigger Use Case (Event B): A different, distinct Splunk event for Customer B (or even a separate event for Customer A) triggers the same use case (name="Malicious IP Observed") at the exact same second as Event A (_time=1771487340).
5. Observe Incorrect Behavior: Instead of creating a new incident for Customer B, the Splunk Add-On updates the existing incident (originally created for Customer A) with information from Event B.

Expected Behavior

For each distinct Splunk security event that triggers an OpenCTI incident creation, a new and unique OpenCTI incident should be created, regardless of whether other events with the same use case name occur within the same second.

Actual Behavior

When name (Splunk use case name) and created (derived from Splunk's _time field, lacking millisecond precision) are identical for two or more distinct Splunk events, the Splunk Add-On generates the same STIX Incident ID. OpenCTI, upon receiving an object with an existing STIX ID, performs an update operation, leading to the merging of distinct security incidents.

Root Cause Analysis

The issue stems from the deterministic generation of the STIX Incident ID using uuid.uuid5 in the Splunk Add-On's utils.py.

Relevant Code Snippets:

ta_opencti_add_on/stix_converter.py` (line 143, or similar):**

stix_case_incident = CustomObjectCaseIncident(
    id=generate_case_incident_id(alert_params.get("name"), event_date), # <--- ID generated here
    name=alert_params.get("name"),
    description=alert_params.get("description"),
    severity=alert_params.get("severity"),
    priority=alert_params.get("priority"),
    labels=alert_params.get("labels"),
    created=event_date,
    external_references=[],
    created_by_ref=stix_author.id,
    object_marking_refs=[marking_id],
    object_refs=observable_ref_ids
)

ta_opencti_add_on/utils.py (line 143, or similar):

def generate_incident_id(name: str, created):
    """
    :param name:
    :param created:
    :return:
    """
    if isinstance(created, datetime.datetime):
        created = created.isoformat() # <--- Converts to ISO format, but _time often lacks milliseconds
    data = {"name": name.lower().strip(), "created": created}
    data = canonicalize(data, utf8=False)
    entity_id = str(uuid.uuid5(uuid.UUID("00abedb4-aa42-466c-9c01-fed23315a9b7"), data)) # <--- Deterministic UUID
    return "incident--" + entity_id

The name parameter typically corresponds to the Splunk alert/use case name, which can be identical across many events. The created parameter is derived from Splunk's _time field, which is often an epoch timestamp without millisecond precision. Consequently, if two distinct events share the same name and occur within the same second, the data dictionary passed to uuid.uuid5 will be identical, leading to the generation of the exact same STIX ID (incident--...).
Impact

Data Integrity Issues: Information from separate security events is merged, leading to an incomplete or inaccurate representation of each incident.
Operational Inefficiency: SOC/CSIRT analysts might miss critical details or misinterpret incidents due to merged data.
Misleading Metrics: Incident counts and resolution times can be skewed.
Compliance Risk: Inability to accurately track and report on distinct security events.

Proposed Solutions / Enhancements

1. Introduce a Configuration Option for Incident Creation Strategy:
   Add a new configuration parameter to the Splunk Add-On (e.g., INCIDENT_CREATION_STRATEGY) that allows users to define the desired behavior:
   ALWAYS_CREATE_NEW (default/recommended for most use cases)
   UPSERT (current behavior, for specific scenarios where updating by name+time is desired)

2. Modify generate_incident_id for Unique ID Generation (for ALWAYS_CREATE_NEW strategy):
   When the strategy is ALWAYS_CREATE_NEW, the generate_incident_id function must incorporate a truly unique identifier from the Splunk event to ensure a unique STIX ID for each distinct event.

Option A (Splunk Event ID): If Splunk provides a unique event ID (e.g., _id field in some contexts), this should be included in the data dictionary for uuid.uuid5.
Option B (High-Precision Timestamp + Random Salt): If a unique Splunk event ID is not reliably available or accessible, modify the function to include a high-precision timestamp (if available from Splunk, e.g., _time with microseconds if possible to extract) combined with a randomly generated UUID or string.

`import uuid
import datetime

def generate_incident_id_unique(name: str, created: datetime.datetime, splunk_event_id: str = None):
if isinstance(created, datetime.datetime):
# Use ISO format with microseconds for better granularity
created_str = created.isoformat(timespec='microseconds')
else:
created_str = created

data = {"name": name.lower().strip(), "created": created_str}

if splunk_event_id:
    data["splunk_event_id"] = splunk_event_id
else:
    # Fallback to a random UUID if no specific Splunk ID is available,
    # ensuring uniqueness even if name and created_str are identical
    data["unique_salt"] = str(uuid.uuid4())

# Ensure canonicalization for consistent UUID5 generation
data = canonicalize(data, utf8=False)
entity_id = str(uuid.uuid5(uuid.UUID("00abedb4-aa42-466c-9c01-fed23315a9b7"), data))
return "incident--" + entity_id`

ref: OCTI2-1984

[CTIBurn0ut]

Root cause is correct — but the fix should use rid, not a random salt

The analysis here is accurate: generate_incident_id/generate_case_incident_id seed uuid5 on {name, created} only, created is derived from _time (frequently second-resolution for notable/tstats/summary-indexed data), so two distinct events sharing a use-case name and the same second produce a byte-identical STIX ID and OpenCTI upserts them into one object. The MSSP-at-volume scenario is exactly where this bites. Note the bug lives in both generate_incident_id and generate_case_incident_id (the case-incident path), so any fix must cover both.

Pushback on the two proposed strategies

Reframing the axis is worth doing. The real choice isn't ALWAYS_CREATE_NEW vs UPSERT — it's deterministic-on-a-unique-event-key vs non-deterministic-random:

• Option B (uuid4() salt) breaks idempotency. Splunk alert actions can fire more than once (retries, scheduler double-fires). A random salt means every re-delivery of the same event mints a brand-new incident — trading a merge bug for a duplicate-flooding bug, which at MSSP volume is arguably worse.
• Option A (a unique Splunk event ID) is the right direction, but it doesn't need a new config field or an author-supplied identifier.

The elegant fix: seed with rid

The Common Action Model framework (cim_actions.py → ModularAction) already guarantees a per-result identifier. ModularAction.update() raises InvalidResultID if a result has no rid — so every result row processed by a CAM-compliant alert action is guaranteed to carry one. sid (search ID) is likewise parsed from the modular-alert payload.

rid is the correct key because it maps to event identity, not execution identity. One result row = one event. Today the seed keys on what the detection is about (name + created); rid restores keying on which event it is.

Proposed seed:

seed = name + created + rid

Why this threads the needle:

• Fixes the collision — 50 distinct notables in one firing → 50 distinct rids → 50 distinct incidents. No over-merge.
• Stays retry-idempotent — a scheduled search re-running over the same window produces the same rows in the same order → same rids → same STIX IDs → OpenCTI upserts onto the existing incidents. No duplicates. Deduping on rid means deduping on actual events, so retry-safety becomes a feature rather than something we sacrifice.

This is zero-config, requires no README change, and uses a framework-guaranteed field — no dependency on the search author emitting _cd/event_id/sourcetype/etc. (Host/sourcetype were considered and rejected: in the exact colliding case the two events typically share those values, so they add no entropy where it's needed.)

Why not sid?

sid changes on every execution, so adding it makes every firing a new incident — reintroducing the duplicate-on-retry exposure. Hold sid back as a deliberate opt-in for teams that explicitly want "every firing is a new detection" semantics, not the default.

Residual edge case (one-line caveat, not a blocker)

If two genuinely different searches — or the same search over a different window — emit a row landing at the same name + created + rid, they'd still merge. This requires identical name, identical second-resolution timestamp, and identical row ordinal across separate result sets — vanishingly rare. If it ever surfaces, sid is the escape hatch, with eyes open about retries.

Implementation sketch

• Thread rid (available on the ModularAction instance / present in the result dict) into generate_incident_id and generate_case_incident_id, folded into the canonicalized seed dict.
• Single call-site change in stix_converter.py to pass it through.

Happy to open a PR once maintainers sign off on the rid semantics and the decision to keep sid opt-in.

Note: this same logic exists verbatim in the newer OpenCTI for Splunk Enterprise add-on (splunk-enterprise-add-on) — both generate_incident_id and generate_case_incident_id carry the identical seed. A companion issue is being filed there so the fix lands in both apps.

[KhalilFilgran]

Hello Team,

Do we have any ETA for the bug fix ?