System test generate should always have same output

[P1llus]

Currently, when generating sample_event.json with elastic-package system --generate it seems to use somewhat random event for its sample.
This is a minor issue because that means the sample event in the integration documentation will constantly change which might confuse some users.
This is specially noticeable if a package has multiple input types (log/tcp/udp), one example would be the fireeye package. While the initial PR might have samples related to TCP, it seems generating it multiple more times will have an example related to LOG.

Do we know if the order the containers are tested is random? How is the event sample picked for the sample_event.json?

[mtojek]

Hey @P1llus ! Could you please describe where is the idempotence lost? Is it about the order of fields or meta properties?

[P1llus]

Its about the whole event actually. System tests sometimes have multiple inputs + multiple lines we test with for each input.
Its not exactly the metadata that changes, but lets say that one time the sample might be from input test #1, and other times its another.

For example zoom has system tests for http and https, while others might have both UDP and TCP.

A follow up question from my side might be, how is the event that is used for sample_event.json picked? Is it from the first or last input tested, the first or last event from an input etc?

[mtojek]

I'm looking at the source code:

elastic-package/internal/testrunner/runners/system/runner.go

Line 454 in 541ee78

if err := writeSampleEvent(dsPath, docs[0]); err != nil {

It means that this is the first event collected for the data stream test.

Its not exactly the metadata that changes, but lets say that one time the sample might be from input test #1, and other times its another.

Hm.. I missed this, does it mean that the system test runner runs tests per data stream in random order?

[P1llus]

If its from the first event, how does this behave when testing multiple inputs? Is it the first document from the first input test? Could the order of the container tests be random at some point? Or does it follow a specific order somewhere?