S3 output retries ignore retry_limit #10819
Description
Bug Report
Describe the bug
Retires for S3 output ignore retry_limit setting and instead retry indefinitely when there is issue with the bucket. OpenTelemetry correctly stops retries after set amount.
To Reproduce
retry_limit for s3 and openTelemetry output is set to fixed numer. S3 bucket name is set to incorrect value, OpenTelemetry port is set to wrong value.
Example output config:
- name: s3
match: '*'
retry_limit: 5
bucket: bucket-bad
region: region
endpoint: endpoint
- name: opentelemetry
match: '*'
retry_limit: 5
host: host
port: bad-port
As a result any log that is routed to s3 output is indefinitely retried every 5 seconds, but log to openTelemetry stops retrying after 5 attempts.
[2025/09/02 10:10:59] [debug] [aws_client] ***: http_do=0, HTTP Status: 404
[2025/09/02 10:10:59] [debug] [upstream] KA connection #68 to ***:443 is now available
[2025/09/02 10:10:59] [debug] [aws_client] Unable to parse API response- response is not valid JSON.
[2025/09/02 10:10:59] [debug] [output:s3:s3.1] PutObject http status=404
[2025/09/02 10:10:59] [debug] [aws] Could not find '<Message>' tag in API response
[2025/09/02 10:10:59] [error] [output:s3:s3.1] PutObject API responded with error='NoSuchBucket'
[2025/09/02 10:10:59] [error] [output:s3:s3.1] Raw PutObject response: HTTP/1.1 404 Not Found
Date: Tue, 02 Sep 2025 10:10:59 GMT
Content-Type: application/xml
Content-Length: 248
Connection: keep-alive
x-amz-request-id: tx000007430e3eb9ac523ff-0068b6c2b3-106608a0f-default
Accept-Ranges: bytes
<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><BucketName>bucket-bad</BucketName><RequestId>tx000007430e3eb9ac523ff-0068b6c2b3-106608a0f-default</RequestId><HostId>106608a0f-default-default</HostId></Error>
[2025/09/02 10:10:59] [error] [output:s3:s3.1] PutObject request failed
[2025/09/02 10:10:59] [error] [output:s3:s3.1] Could not send chunk with tag tag
[2025/09/02 11:09:28] [ warn] [engine] failed to flush chunk '39-1756811270.9104823.flb', retry in 10 seconds: task_id=0, input=emitter_for_rewrite_tag.1 > output=opentelemetry.0 (out_id=0)
[2025/09/02 11:09:49] [error] [upstream] connection #54 to tcp://***:444 timed out after 10 seconds (connection timeout)
[2025/09/02 11:09:49] [debug] [net] TCP connection timed out: ***:444
[2025/09/02 11:09:49] [debug] [net] could not connect to ***:444
[2025/09/02 11:09:49] [debug] [upstream] connection #54 failed to ***:444
[2025/09/02 11:09:49] [debug] http session creation error
[2025/09/02 11:09:49] [error] [output:opentelemetry:opentelemetry.0] error initializing http request
[2025/09/02 11:09:49] [debug] [out flush] cb_destroy coro_id=5
[2025/09/02 11:09:49] [debug] [task] task_id=0 reached retry-attempts limit 5/5
[2025/09/02 11:09:49] [error] [engine] chunk '39-1756811270.9104823.flb' cannot be retried: task_id=0, input=emitter_for_rewrite_tag.1 > output=opentelemetry.0
[2025/09/02 11:09:49] [debug] [task] destroy task=0x7f06d9229be0 (task_id=0)
Expected behavior
- After set number of retries (5 in the example above), the retries to s3 output should stop.
- All outputs should behave in the same, predictable way.
Your Environment
- Version used: 4.0.8