Add AwsBatchLogProcessor and OtlpAwsLogExporter Logs Pipeline

[liustve]

In order to support CW Logs implementation for handling LLO and adhering to their new limits in ADOT, they specifically indicated that if the log is an LLO then we would export immediately:

1. For each new log record:
    1. If (buffer_size + record_size) < 1MB: Add to buffer
    2. If (buffer_size + record_size) > 1MB: Ingest current buffer, then add new record to empty buffer
2. If buffer_size > 1MB: Ingest immediately
3. Repeat from step 1

Furthermore they need a new header for the log object that contains all of the paths to the String leaf nodes in the log body represented in JSON protobuf format. Below is a representation of the consolidated Gen AI event log schema in JSON protobuf format.

Description of changes:

1. Adds AwsBatchLogRecordProcessor a backwards compatible custom logs BatchProcessor which has the following invariants:

   • The unserialized, uncompressed data size of exported batches will ALWAYS be <= 1 MB except for the case below:
   • If the data size of an exported batch is ever > 1 MB then the batch size is always length 1

2. Added support for estimating the size of a log record based on its body and attributes. However, because logs are represented as a self-referential data-type, the log record can potentially be very nested and complex. In order to lower the impact on performance when calculating the size of a log, a depth limit is added to how deep the log will be searched.

3. OTLPAwsLogExporter: Adds a new behavior for Retry delay based on server-side response of Retry-After header. Injects the LLO header flag if the size of the exported data > 1 MB.

4. Customize the auto instrumentation to use the new AwsBatchLogRecordProcessor

Testing

• Unit tests for AwsBatchLogRecordProcessor to support the new LLO handling logic
• Unit tests for OtlpAwsLogExporter to validate the new Retry and LLO header logic
• Manual E2E testing to verify that the logic for batching and LLO path header works for Large Logs (mocked the size of the log by setting the log size to 1,000,000 bytes)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

    {
        "body": {
            "kvlistValue": {
                "values": [
                    {
                        "key": "input",
                        "value": {
                            "kvlistValue": {
                                "values": [
                                    {
                                        "key": "messages",
                                        "value": {
                                            "arrayValue": {
                                                "values": [
                                                    {
                                                        "kvlistValue": {
                                                            "values": [
                                                                {
                                                                    "key": "content",
                                                                    "value": {
                                                                        "stringValue": "hello"
                                                                    }
                                                                },
                                                                {
                                                                    "key": "role",
                                                                    "value": {
                                                                        "stringValue": "user"
                                                                    }
                                                                }
                                                            ]
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    }
                                ]
                            }
                        }
                    },
                    {
                        "key": "output",
                        "value": {
                            "kvlistValue": {
                                "values": [
                                    {
                                        "key": "messages",
                                        "value": {
                                            "arrayValue": {
                                                "values": [
                                                    {
                                                        "kvlistValue": {
                                                            "values": [
                                                                {
                                                                    "key": "content",
                                                                    "value": {
                                                                        "stringValue": "hi"
                                                                    }
                                                                },
                                                                {
                                                                    "key": "role",
                                                                    "value": {
                                                                        "stringValue": "assistant"
                                                                    }
                                                                }
                                                            ]
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    }
                                ]
                            }
                        }
                    }
                ]
            }
        }
    }

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.