Feature/windows event id filtering #1737

Paamicky · 2025-06-18T21:16:17Z

Description of the issue

The CloudWatch Agent currently filters Windows Event Logs based on security levels (i.e Error, Critical, Information...). This does not allow customers to only collect specific and relevant logs to CloudWatch

Description of changes

Add Event ID filtering to the CWAgent Windows logs collection

sample configuration of event ID filtering

  "logs": {
    "logs_collected": {
      "windows_events": {
        "collect_list": [
          {
            "event_name": "System",
            "event_ids": [4624, 4625, 4634],
            "log_group_name": "WindowsEventIds-Test",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}

Caution

This PR has a testing PR linked to here: aws/amazon-cloudwatch-agent-test#541, merge this first.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Added unit tests for event ID validation and processing
Manually tested on Windows to verify events are properly filtered by ID
Verified backward compatibility with existing configurations

Requirements

Before commiting your code, please do the following steps.

Run make fmt and make fmt-sh
Run make lint

Integration Tests

To run integration tests against this PR, add the ready for testing label.

plugins/inputs/windows_event_log/windows_event_log.go

plugins/inputs/windows_event_log/wineventlog/wineventlog.go

plugins/inputs/windows_event_log/wineventlog/wineventlog_test.go

dricross · 2025-06-23T13:37:58Z

translator/tocwconfig/sampleConfig/windows_eventlog_only_config.conf

@@ -24,6 +24,7 @@
      batch_read_size = 170
      event_levels = ["4", "0", "2"]
      event_name = "System"
+      log_group_class = ""


Do we know why this is being added? I don't think your changes should have affected this field.

Yes, I think I raised this earlier, I don't know why it is been added. I'm thinking it will be from a previous update to the agent from another commit.

Would be good to find that commit and understand why this change is necessary more concretely before we merge this in. I am especially interested in the region_type field under [[outputs.cloudwatchlogs]]

I will check that.

The log_group_class and region_type were implemented about 20 months ago by previous commits. It got added since I'm generating the TOML files for the first time.

You can find more about it here:
log_group_class

Region_type:
rurleRegion
regionType

translator/tocwconfig/tocwconfig_test.go

dricross · 2025-06-23T13:54:11Z

translator/translate/logs/logs_collected/windows_events/collect_list/collectlist.go

@@ -118,4 +119,41 @@ func addFixedJsonConfig(result map[string]interface{}) {
 		}


this is comment is for line 100. I can't comment on it directly though since it wasn't modified.

The function should not return early anymore if the EventLevelsKey is not found since the event_levels key is no longer strictly necessary. As this function is right now, I believe it won't validate the event IDs if the event_levels key is not present. You'll need to change the event levels to act like what you have in the event ID section.

everything related to event levels needs to go inside the if statement or there will be some behavior changes:

if eventLevels, ok := result[EventLevelsKey]; ok { inputEventLevels = eventLevels.([]interface{}) resultEventLevels := []interface{}{} for _, eventLevel := range inputEventLevels { switch eventLevel.(string) { case "CRITICAL": resultEventLevels = append(resultEventLevels, "1") case "ERROR": resultEventLevels = append(resultEventLevels, "2") case "WARNING": resultEventLevels = append(resultEventLevels, "3") case "INFORMATION": resultEventLevels = append(resultEventLevels, "4", "0") case "VERBOSE": resultEventLevels = append(resultEventLevels, "5") default: translator.AddErrorMessages(GetCurPath(), fmt.Sprintf("Cannot find the mapping for Windows event level %v.", eventLevel)) } } result[EventLevelsKey] = resultEventLevels }

dricross · 2025-06-23T13:55:48Z

translator/translate/logs/logs_collected/windows_events/collect_list/collectlist.go

+	const (
+		minEventID = 0
+		maxEventID = 65535
+	)


would be good to link to documentation or explain why these values are selected

we should also have test cases for these values and their neighbors (e.g. -1, 0, 1, 65534, 65535, 65536) to ensure they are all handled properly

Added link to documentation!

Yes that's right, but technically the event ids ranges from 0-65535. So I thought for validation or programming purposes, it would be correct as these are the absolute min and max values that can be stored in the event ids field. They are mostly reserved for if application crashes or fails to install,,

translator/translate/logs/logs_collected/windows_events/collect_list/collectlist_test.go

cmd/config-translator/translator_test.go

okankoAMZ · 2025-06-23T20:14:04Z

plugins/inputs/windows_event_log/windows_event_log.go

@@ -61,6 +61,7 @@ func (s *Plugin) SampleConfig() string {
 	[[inputs.windows_event_log.event_config]]
 	event_name = "System"
 	event_levels = ["2", "3"]
+	event_ids = [1001, 1002]


What are these event ids what are they representing, can we comment that here?

They are generic event ids to show how the final sampleConfig of the Toml will look like. it serves as documentation and example config for users of the Windows event Log plugin. So I just add the event id since it is a feature now.

okankoAMZ · 2025-06-23T20:22:12Z

plugins/inputs/windows_event_log/wineventlog/utils.go

@@ -36,34 +25,7 @@ const (
 	UNKNOWN     = "UNKNOWN"
 )

-var NumberOfBytesPerCharacter = UnknownBytesPerCharacter
-
-func RenderEventXML(eventHandle EvtHandle, renderBuf []byte) ([]byte, error) {


why are we deleting this function can you explain in your PR description?

I just moved them to a different file due to the windows build tags. I wanted run some test on the linux environment so it doesn't skip those tests.

okankoAMZ · 2025-06-23T20:22:16Z

plugins/inputs/windows_event_log/wineventlog/utils.go

-	return utf16ToUTF8Bytes(renderBuf, bufferUsed)
-}
-
-func CreateBookmark(channel string, recordID uint64) (h EvtHandle, err error) {


why are we deleting this function can you explain in your PR description?

Moved to a different file due to the windows build tags.

okankoAMZ · 2025-06-23T20:22:28Z

plugins/inputs/windows_event_log/wineventlog/utils.go


-func WindowsEventLogLevelName(levelId int32) string {


why are we deleting this function can you explain in your PR description?

Moved to a different file due to the windows build tags.

okankoAMZ · 2025-06-23T20:23:01Z

plugins/inputs/windows_event_log/wineventlog/utils_windows.go

+
+var NumberOfBytesPerCharacter = UnknownBytesPerCharacter
+
+func RenderEventXML(eventHandle EvtHandle, renderBuf []byte) ([]byte, error) {


Okay so you are moving into a different file. Please document that in your description

Alright sure

okankoAMZ · 2025-06-23T20:24:16Z

translator/tocwconfig/sampleConfig/windows_eventlog_only_config.conf

    region = "us-west-2"
+    region_type = "ACJ"


what is region type and what is "ACJ"?

yeah I will investigate, it got added when I generated the files, probably from a previous commit.

The region_type was implemented about 20 months ago by previous commits. It got added since I'm generating the TOML files for the first time.

You can find more about it here:

Region_type:
rurleRegion
regionType

okankoAMZ · 2025-06-23T20:25:33Z

translator/tocwconfig/totomlconfig/toTomlConfig.go

 	if err != nil {
 		log.Panicf("Encode to a valid TOML config fails because of %v", err)
 	}
 	return buf.String()
 }
+
+// ensure integers in arrays are preserved
+func processValue(val interface{}) interface{} {


shouldn't we have some kind of validation and returning error here. With the current implementation it seems like this function could never fail, is this true?

okankoAMZ · 2025-06-23T20:42:54Z

translator/translate/logs/logs_collected/windows_events/collect_list/collectlist.go

-	if eventLevels, ok := result[EventLevelsKey]; !ok {
-		return
-	} else {


With this change now, we keep going when the eventLevel is missing. Is this intentional?

if the eventLevel is missing it will proceed to check the for eventids, actually this still sets eventlevels to an empty slice of string instead of nil, I will fix this.

dricross · 2025-06-23T20:37:34Z

translator/translate/logs/logs_collected/windows_events/collect_list/collectlist.go

+	const (
+		minEventID = 0
+		maxEventID = 65535
+	)


dricross · 2025-06-23T20:38:38Z

translator/translate/logs/logs_collected/windows_events/windows_event_test.go

@@ -25,6 +25,10 @@ func TestApplyRule(t *testing.T) {
              "INFORMATION",
              "SUCCESS"
            ],
+			"event_ids": [


nit: formatting

dricross · 2025-06-23T20:39:32Z

translator/tocwconfig/totomlconfig/toTomlConfig.go

+	// Process value to ensure integers in arrays are preserved
+	processedVal := processValue(val)


Interesting. There are some other integers that aren't mangled like log retention. I wonder why those work. Perhaps because they aren't in an array? We should understand why those integers don't require additional processing.

dricross · 2025-06-23T20:40:18Z

translator/tocwconfig/sampleConfig/windows_eventlog_only_config.conf

@@ -24,6 +24,7 @@
      batch_read_size = 170
      event_levels = ["4", "0", "2"]
      event_name = "System"
+      log_group_class = ""


Would be good to find that commit and understand why this change is necessary more concretely before we merge this in. I am especially interested in the region_type field under [[outputs.cloudwatchlogs]]

dricross · 2025-06-23T20:45:09Z

translator/translate/logs/logs_collected/windows_events/collect_list/collectlist.go

@@ -118,4 +119,41 @@ func addFixedJsonConfig(result map[string]interface{}) {
 		}


everything related to event levels needs to go inside the if statement or there will be some behavior changes:

if eventLevels, ok := result[EventLevelsKey]; ok { inputEventLevels = eventLevels.([]interface{}) resultEventLevels := []interface{}{} for _, eventLevel := range inputEventLevels { switch eventLevel.(string) { case "CRITICAL": resultEventLevels = append(resultEventLevels, "1") case "ERROR": resultEventLevels = append(resultEventLevels, "2") case "WARNING": resultEventLevels = append(resultEventLevels, "3") case "INFORMATION": resultEventLevels = append(resultEventLevels, "4", "0") case "VERBOSE": resultEventLevels = append(resultEventLevels, "5") default: translator.AddErrorMessages(GetCurPath(), fmt.Sprintf("Cannot find the mapping for Windows event level %v.", eventLevel)) } } result[EventLevelsKey] = resultEventLevels }

Paamicky and others added 7 commits June 6, 2025 13:23

Add event ID filtering for windows logs to CWAgent

cd878d0

mend

f6d5cdf

mend

119c7d7

mend

de9a111

mend

037fe7f

update just for integ test

5dd88d6

Merge branch 'main' into feature/windows-eventId-filtering

144fac4

Paamicky force-pushed the feature/windows-eventId-filtering branch 3 times, most recently from b758b9e to 516df4e Compare June 20, 2025 16:09

fix: resolve workflow lint issues

2276ab2

Paamicky force-pushed the feature/windows-eventId-filtering branch from 516df4e to 2276ab2 Compare June 20, 2025 17:42

Paamicky marked this pull request as ready for review June 23, 2025 01:09

Paamicky requested a review from a team as a code owner June 23, 2025 01:09

Paamicky requested review from jefchien and okankoAMZ June 23, 2025 01:11

dricross reviewed Jun 23, 2025

View reviewed changes

Paamicky force-pushed the feature/windows-eventId-filtering branch from 9d99660 to 2667175 Compare June 23, 2025 19:27

okankoAMZ reviewed Jun 23, 2025

View reviewed changes

dricross reviewed Jun 23, 2025

View reviewed changes

Merge branch 'main' into feature/windows-eventId-filtering

ea8ce75

Paamicky force-pushed the feature/windows-eventId-filtering branch from 2667175 to ea8ce75 Compare June 24, 2025 03:03

		@@ -118,4 +119,41 @@ func addFixedJsonConfig(result map[string]interface{}) {
		}


		var NumberOfBytesPerCharacter = UnknownBytesPerCharacter

		func RenderEventXML(eventHandle EvtHandle, renderBuf []byte) ([]byte, error) {

		// Process value to ensure integers in arrays are preserved
		processedVal := processValue(val)

Feature/windows event id filtering #1737

Are you sure you want to change the base?

Feature/windows event id filtering #1737

Uh oh!

Conversation

Paamicky commented Jun 18, 2025 • edited by okankoAMZ Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of the issue

Description of changes

License

Tests

Requirements

Integration Tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Paamicky Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Paamicky Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Paamicky Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Paamicky commented Jun 18, 2025 •

edited by okankoAMZ

Loading

Paamicky Jun 24, 2025 •

edited

Loading

Paamicky Jun 24, 2025 •

edited

Loading

Paamicky Jun 24, 2025 •

edited

Loading