Skip to content

[aws_logs] fix number_of_workers to be always passed in hbs template #14416

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
+18 −11
Open

[aws_logs] fix number_of_workers to be always passed in hbs template #14416

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
95161a8
[aws_logs] fix number_of_workers to be always passed in hbs template
mykola-elastic Jul 3, 2025
bc2393d
update manifest.yml, changelog.yml
mykola-elastic Jul 3, 2025
7f45e7a
add comment
mykola-elastic Jul 3, 2025
994b9a6
mention `number_of_workers` in the README
mykola-elastic Jul 10, 2025
b674922
remove unneeded comment
mykola-elastic Jul 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/aws_logs/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ expensive in terms of performance and costs and should be preferably used only
when no SQS notification can be attached to the S3 buckets. This input
integration also supports S3 notification from SNS to SQS.

You can enable SQS notification method by setting `queue_url` configuration value.
You can enable SQS notification method by setting `queue_url` and `number_of_workers` configuration values.
You can enable S3 bucket list polling method by setting `bucket_arn`, `access_point_arn`
or `non_aws_bucket_name` configuration values and `number_of_workers` value.

Expand Down
5 changes: 5 additions & 0 deletions packages/aws_logs/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
- version: "1.8.1"
changes:
- description: Fix handling of SQS worker count configuration.
type: bugfix
link: https://github.com/elastic/integrations/pull/14416
- version: "1.8.0"
changes:
- description: Add support to configure start_timestamp and ignore_older configurations for AWS S3 backed inputs.
Expand Down
7 changes: 4 additions & 3 deletions packages/aws_logs/data_stream/generic/agent/stream/aws-cloudwatch.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,6 @@ log_group_name_prefix: {{ log_group_name_prefix }}
{{#if include_linked_accounts_with_prefix }}
include_linked_accounts_for_prefix_mode: {{ include_linked_accounts_with_prefix }}
{{/if}}
{{#if number_of_workers }}
number_of_workers: {{ number_of_workers }}
{{/if}}
{{/unless}}
{{/unless}}

Expand Down Expand Up @@ -50,6 +47,10 @@ log_stream_prefix: {{ log_stream_prefix }}
{{/if}}
{{/unless}}

{{#if number_of_workers }}
number_of_workers: {{ number_of_workers }}
{{/if}}

{{#if start_position }}
start_position: {{ start_position }}
{{/if}}
Expand Down
7 changes: 4 additions & 3 deletions packages/aws_logs/data_stream/generic/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,6 @@ When using an S3 bucket, you can specify only one of the following options:
}}

{{! shared S3 bucket polling options }}
{{#if number_of_workers }}
number_of_workers: {{ number_of_workers }}
{{/if}}

{{#if start_timestamp}}
start_timestamp: {{start_timestamp}}
Expand Down Expand Up @@ -76,6 +73,10 @@ access_point_arn: {{ access_point_arn }}

{{/unless}}{{! end S3 bucket polling }}

{{! allows number of workers to be configured for SQS queue and S3 buckets}}
{{#if number_of_workers }}
number_of_workers: {{ number_of_workers }}
{{/if}}
{{#if buffer_size }}
buffer_size: {{ buffer_size }}
{{/if}}
Expand Down
4 changes: 2 additions & 2 deletions packages/aws_logs/data_stream/generic/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,12 +179,12 @@ streams:
description: ARN of the AWS S3 Access Point that will be polled for list operation. (This is an alternative to the Bucket ARN, and required when `queue_url`, `bucket_arn` or `non_aws_bucket_name` are not set).
- name: number_of_workers
type: integer
title: Number of Workers
title: "[S3/SQS] Number of Workers"
multi: false
required: false
default: 1
show_user: true
description: Number of workers that will process the S3 objects listed. (Required when `bucket_arn` or `access_point_arn` are set).
description: Number of workers that will process the S3 objects listed.
- name: start_timestamp
type: text
title: "Start Timestamp"
Expand Down
2 changes: 1 addition & 1 deletion packages/aws_logs/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ expensive in terms of performance and costs and should be preferably used only
when no SQS notification can be attached to the S3 buckets. This input
integration also supports S3 notification from SNS to SQS.

You can enable SQS notification method by setting `queue_url` configuration value.
You can enable SQS notification method by setting `queue_url` and `number_of_workers` configuration values.
You can enable S3 bucket list polling method by setting `bucket_arn`, `access_point_arn`
or `non_aws_bucket_name` configuration values and `number_of_workers` value.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we dont make any reference to number of workers. You can add it back in the doc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gizas, I've returned that and also mentioned it in the sentence about SQS too. Good?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elastic/obs-ds-hosted-services do you need this fix? Issue: #14403

@mykola-elastic sorry confused, do you mean to mark 14403 as done when this PR will be merged? Personally have not verified it e2e

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gizas what would be the correct process for this?
Shall e2e testing be performed and described in this PR? I guess showing the proof that the number of workers is actually changed..

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing the setting Number of Workers does not have any effect, the integration still starts using the default number of 5 workers for the SQS input.

If you can showcase that we have a new value after the change we are ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gizas I did the following:
Set number_of_workers to 6 for S3 and 7 for Cloudwatch.
Also set queue_url for S3 and log_group_arn for Cloudwatch.

The policy before this PR (See that there are no number_of_workers here):

id: elastic-agent-managed-ep
revision: 3
outputs:
  default:
    type: elasticsearch
    hosts:
      - https://elasticsearch:9200
    ssl.ca_trusted_fingerprint: 57E00CA616A85926934E861AC4731978FCA0E33BB519DCB8AEBD61CCF4F7F7AC
    preset: latency
fleet:
  hosts:
    - https://fleet-server:8220
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices: []
    _elastic_agent_checks:
      cluster:
        - monitor
    39a03b08-6f90-4959-8c4c-32b8366d1e2a:
      indices:
        - names:
            - logs-*-*
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-*-*
          privileges:
            - auto_configure
            - create_doc
agent:
  download:
    sourceURI: https://artifacts.elastic.co/downloads/
  monitoring:
    enabled: false
    logs: false
    metrics: false
    traces: false
  features: {}
  protection:
    enabled: false
    uninstall_token_hash: 9G7sLqRFHP1Joeq0VAX6pZe1q8Mo4BTIbhDHQD/R/ow=
    signing_key: >-
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEZ0xeikdK85bTrAq6JTQMmyAPO9dQIl5CxKtzMuk7mitXZFxm8tZwMUf76WHG+1zRmwpNzgkWUnDSxTTAaPbj8Q==
inputs:
  - id: aws-s3-aws_logs-39a03b08-6f90-4959-8c4c-32b8366d1e2a
    name: aws_logs-1
    revision: 2
    type: aws-s3
    use_output: default
    meta:
      package:
        name: aws_logs
        version: 1.8.0
    data_stream:
      namespace: default
    package_policy_id: 39a03b08-6f90-4959-8c4c-32b8366d1e2a
    streams:
      - id: aws-s3-aws_logs.generic-39a03b08-6f90-4959-8c4c-32b8366d1e2a
        data_stream:
          dataset: aws_logs.generic
        queue_url: http://test
        max_bytes: 10MiB
        max_number_of_messages: 5
        sqs.max_receive_count: 5
        sqs.wait_time: 20s
        file_selectors: null
        tags:
          - forwarded
        publisher_pipeline.disable_host: true
        parsers: null
  - id: aws-cloudwatch-aws_logs-39a03b08-6f90-4959-8c4c-32b8366d1e2a
    name: aws_logs-1
    revision: 2
    type: aws-cloudwatch
    use_output: default
    meta:
      package:
        name: aws_logs
        version: 1.8.0
    data_stream:
      namespace: default
    package_policy_id: 39a03b08-6f90-4959-8c4c-32b8366d1e2a
    streams:
      - id: aws-cloudwatch-aws_logs.generic-39a03b08-6f90-4959-8c4c-32b8366d1e2a
        data_stream:
          dataset: aws_logs.generic
        log_group_arn: test-log-group-arn
        start_position: beginning
        scan_frequency: 1m
        api_sleep: 200ms
        tags:
          - forwarded
        publisher_pipeline.disable_host: true
signed:
  data: >-
    eyJpZCI6ImVsYXN0aWMtYWdlbnQtbWFuYWdlZC1lcCIsImFnZW50Ijp7ImZlYXR1cmVzIjp7fSwicHJvdGVjdGlvbiI6eyJlbmFibGVkIjpmYWxzZSwidW5pbnN0YWxsX3Rva2VuX2hhc2giOiI5RzdzTHFSRkhQMUpvZXEwVkFYNnBaZTFxOE1vNEJUSWJoREhRRC9SL293PSIsInNpZ25pbmdfa2V5IjoiTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFWjB4ZWlrZEs4NWJUckFxNkpUUU1teUFQTzlkUUlsNUN4S3R6TXVrN21pdFhaRnhtOHRad01VZjc2V0hHKzF6Um13cE56Z2tXVW5EU3hUVEFhUGJqOFE9PSJ9fSwiaW5wdXRzIjpbeyJpZCI6ImF3cy1zMy1hd3NfbG9ncy0zOWEwM2IwOC02ZjkwLTQ5NTktOGM0Yy0zMmI4MzY2ZDFlMmEiLCJuYW1lIjoiYXdzX2xvZ3MtMSIsInJldmlzaW9uIjoyLCJ0eXBlIjoiYXdzLXMzIn0seyJpZCI6ImF3cy1jbG91ZHdhdGNoLWF3c19sb2dzLTM5YTAzYjA4LTZmOTAtNDk1OS04YzRjLTMyYjgzNjZkMWUyYSIsIm5hbWUiOiJhd3NfbG9ncy0xIiwicmV2aXNpb24iOjIsInR5cGUiOiJhd3MtY2xvdWR3YXRjaCJ9XX0=
  signature: >-
    MEUCIQC47UBGKnazr/w0d+MuYxmvqHAEq0C/F21aslsCPYU2RgIgCY8hNc4bPFVT3rd2dtQ9hmSy9cn1JLQdK4KGzDT2nic=
secret_references: []
namespaces: []

The policy after this PR (the number_of_workers are now in the policy):

id: elastic-agent-managed-ep
revision: 2
outputs:
  default:
    type: elasticsearch
    hosts:
      - https://elasticsearch:9200
    ssl.ca_trusted_fingerprint: 57E00CA616A85926934E861AC4731978FCA0E33BB519DCB8AEBD61CCF4F7F7AC
    preset: latency
fleet:
  hosts:
    - https://fleet-server:8220
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices: []
    _elastic_agent_checks:
      cluster:
        - monitor
    a3f1c02f-e771-4388-9ce7-96aeb6139a65:
      indices:
        - names:
            - logs-*-*
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-*-*
          privileges:
            - auto_configure
            - create_doc
agent:
  download:
    sourceURI: https://artifacts.elastic.co/downloads/
  monitoring:
    enabled: false
    logs: false
    metrics: false
    traces: false
  features: {}
  protection:
    enabled: false
    uninstall_token_hash: myV+/PC9h7tRznolRUvKDaTX0ppPn8sOuq7FX2oXKso=
    signing_key: >-
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEJspyjkf1F7Vu1HgYTa268flRF4l/IXtOIjvIX+A44OkkOn4FG8XIhnQKFcovgez0oqX96Gt1twRUyR6i4WayHQ==
inputs:
  - id: aws-s3-aws_logs-a3f1c02f-e771-4388-9ce7-96aeb6139a65
    name: aws_logs-1
    revision: 1
    type: aws-s3
    use_output: default
    meta:
      package:
        name: aws_logs
        version: 1.8.1
    data_stream:
      namespace: default
    package_policy_id: a3f1c02f-e771-4388-9ce7-96aeb6139a65
    streams:
      - id: aws-s3-aws_logs.generic-a3f1c02f-e771-4388-9ce7-96aeb6139a65
        data_stream:
          dataset: aws_logs.generic
        queue_url: http://test
        number_of_workers: 6
        max_bytes: 10MiB
        max_number_of_messages: 5
        sqs.max_receive_count: 5
        sqs.wait_time: 20s
        file_selectors: null
        tags:
          - forwarded
        publisher_pipeline.disable_host: true
        parsers: null
  - id: aws-cloudwatch-aws_logs-a3f1c02f-e771-4388-9ce7-96aeb6139a65
    name: aws_logs-1
    revision: 1
    type: aws-cloudwatch
    use_output: default
    meta:
      package:
        name: aws_logs
        version: 1.8.1
    data_stream:
      namespace: default
    package_policy_id: a3f1c02f-e771-4388-9ce7-96aeb6139a65
    streams:
      - id: aws-cloudwatch-aws_logs.generic-a3f1c02f-e771-4388-9ce7-96aeb6139a65
        data_stream:
          dataset: aws_logs.generic
        log_group_arn: test-log-group-arn
        number_of_workers: 7
        start_position: beginning
        scan_frequency: 1m
        api_sleep: 200ms
        tags:
          - forwarded
        publisher_pipeline.disable_host: true
signed:
  data: >-
    eyJpZCI6ImVsYXN0aWMtYWdlbnQtbWFuYWdlZC1lcCIsImFnZW50Ijp7ImZlYXR1cmVzIjp7fSwicHJvdGVjdGlvbiI6eyJlbmFibGVkIjpmYWxzZSwidW5pbnN0YWxsX3Rva2VuX2hhc2giOiJteVYrL1BDOWg3dFJ6bm9sUlV2S0RhVFgwcHBQbjhzT3VxN0ZYMm9YS3NvPSIsInNpZ25pbmdfa2V5IjoiTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFSnNweWprZjFGN1Z1MUhnWVRhMjY4ZmxSRjRsL0lYdE9JanZJWCtBNDRPa2tPbjRGRzhYSWhuUUtGY292Z2V6MG9xWDk2R3QxdHdSVXlSNmk0V2F5SFE9PSJ9fSwiaW5wdXRzIjpbeyJpZCI6ImF3cy1zMy1hd3NfbG9ncy1hM2YxYzAyZi1lNzcxLTQzODgtOWNlNy05NmFlYjYxMzlhNjUiLCJuYW1lIjoiYXdzX2xvZ3MtMSIsInJldmlzaW9uIjoxLCJ0eXBlIjoiYXdzLXMzIn0seyJpZCI6ImF3cy1jbG91ZHdhdGNoLWF3c19sb2dzLWEzZjFjMDJmLWU3NzEtNDM4OC05Y2U3LTk2YWViNjEzOWE2NSIsIm5hbWUiOiJhd3NfbG9ncy0xIiwicmV2aXNpb24iOjEsInR5cGUiOiJhd3MtY2xvdWR3YXRjaCJ9XX0=
  signature: >-
    MEUCIQCVoFSji38/z/VOOsK15OlM6XMSm7sKkbK0lvZVJPw0/QIgRoRq/esTIMI4hyqKxzo21huIH2Rt/SsSsHsUeiypW4c=
secret_references: []
namespaces: []


Expand Down
2 changes: 1 addition & 1 deletion packages/aws_logs/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: aws_logs
title: Custom AWS Logs
description: Collect raw logs from AWS S3 or CloudWatch with Elastic Agent.
type: integration
version: "1.8.0"
version: "1.8.1"
categories:
- cloud
- observability
Expand Down