aws: optimise cloudtrail field retention work

[efd6]

Proposed commit message

aws: optimise cloudtrail field retention work

In #14236 we allowed users to select which extended fields they wanted
to retain in order to reduce storage costs in cases where they did not
what the full set of capacities that the data stream can provide. We
did not however prevent the work of collecting those unwanted fields.
This change does that, avoiding retaining fields that will ultimately
not be kept if possible.

It is unfortunate that the wide variety of fields is needed at all, but
resolving that depends on improving platform support for the diversity
of fields that the data source provides and then making more efficient
use of those improvements in the detection rules. Until then, this is
what we have.

Note

Best reviewed commitwise.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes aws: optimise cloudtrail ingest pipeline for cases where fruitless work is done #14429
• Depends aws: fix agent template typo #14439

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[romulets]

Great work! A few comments, but I really love the consistency changes!

[efd6]

/test

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[Mikaayenson]

@efd6 qq: Can you use the required_fields that are autogenerated and populated with the AWS rules artifact and sent upstream to Kibana (vs statically using a list in the integration)?

[efd6]

It's not clear to me how that would be achieved.

[cpascale43]

Hey @efd6 - if they remove fields will this have an impact on dashboards/rules? If so, can we make a plan to document this somewhere?

[efd6]

@cpascale43 This change does not remove fields. That was already done in #14236. This just changes the time that the removal is done, from being a post facto remove, to a preemptive non-add. The change linked above shows the documentation that is presented to the user.

[efd6]

This change is already starting to rot due to the complexity of the code here.

[efd6]

/test

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
99.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 8a3e413

History

• 💔 Build #28540 failed 1c17b14
• 💔 Build #28537 failed 1c17b14
• 💚 Build #28239 succeeded c9fe96f
• 💚 Build #28236 succeeded b2d6b4e
• 💔 Build #28208 failed b2d6b4e

cc @efd6