Pagination fails if long list passed to filter value

[juliapaluch]

Description

Pagination fails when I pass a long list to the values key in a filter passed to filters.

Environment

1. OS (where OpenCTI server runs): AWS Amazon Linux
2. OpenCTI version: 6.2.18
3. Other environment details: Using Amazon OpenSearch

Reproducible Steps

Run this script, changing what's needed:

client = OpenCTIApiClient() # init your client

custom_attributes = """id"""
filters = {
  "mode": "and",
  "filters": [
    {"key":"creator_id","values":["<creator id>"]},
  ],
  "filterGroups": [],
}

result = client.stix_cyber_observable.list(
  getAll=True,
  customAttributes=custom_attributes,
  filters=filters
)

observable_ids = [observable["id"] for observable in result]

custom_attributes = """id"""
filters = {
  "mode": "and",
  "filters": [
    {"key": "report_types", "values": "<custom-report-type>"},
    {"key": "objects", "values": observable_ids, "operator": "eq", "mode": "or"}
  ],
  "filterGroups": [],
}

result = client.report.list(
  getAll=True,
  customAttributes=custom_attributes,
  filters=filters
)

print(f"Got {len(result)} reports!")

When I run this in our production instance, I get 100,820 results from the call to client.stix_cyber_observable.list, which is then passed to the filters in the call to client.report.list, which is where the exception is thrown. I suspect it has something to do with the length of this list because when I comment out the getAll param in the call to client.stix_cyber_observable.list, which gives me only 100 results, I get the expected output with no error.

Expected Output

I expect to get the number of reports that meet my criteria.

Actual Output

Traceback (most recent call last):
  File "my_script.py", line 42, in <module>
    result = client.report.list(
             ^^^^^^^^^^^^^^^^^^^
  File "/path/to/.venv/lib/python3.12/site-packages/pycti/entities/opencti_report.py", line 538, in list
    result = self.opencti.query(
             ^^^^^^^^^^^^^^^^^^^
  File "/path/to/.venv/lib/python3.12/site-packages/pycti/api/opencti_api_client.py", line 384, in query
    raise ValueError(value_error)
ValueError: {'name': 'DATABASE_ERROR', 'error_message': 'Fail to execute engine pagination'}

[juliapaluch]

If anyone encounters this issue and wants a quick workaround, splitting the filters up into batches of size 100 seems to work:

step = 100
for i in range(0, len(observable_ids), step):
  print(f"Filtering on observables {i} to {i + step}")
  filters["filters"][1]["values"] = observable_ids[i:i+step]

  result = client.report.list(
    getAll=True,
    customAttributes=custom_attributes,
    filters=filters
  )

  reports.extend(result)

Just remember to dedupe at the end 😺

[nino-filigran]

Issue lies at API level, keeping this issue opened to ensure we can protect the API