How do I consolidate application logs across accounts?

[yorinasub17]

I am trying to consolidate our app logs in to the logs account and could use some guidance on how Gruntwork proposes I do that.

We have EKS clusters in various accounts and have fluent-bit installed in the cluster which ships the logs to a CloudWatch Log Group in the account itself. We would want these logs be available in the centralized logs account, so we can look at exposing the logs to developers.

---

Tracked in ticket #109119

[yorinasub17]

This is actually a bit tricky to do since CloudWatch Logs doesn’t natively support cross account access, making it a bit of a headache to consolidate the logs. There are a couple of workarounds, but none of them are particularly easy to implement, and we don't really have modules that make the process any easier.

• Use Lambda and CloudWatch Log Subscription Filters. In this approach, you can create a Lambda function that does the cross account migration process for you. You will want to define a new Lambda function that is able to assume a role in the logs account to push the logs from the app account to a CloudWatch Log Group in the logs account. You can then hook this up to the source Log Group in the app account to stream the logs every time a new entry is pushed to the app account log group.

• Use Kinesis to ship log data to the logs account. In this approach, you can configure fluent-bit to ship the logs to an AWS Kinesis stream instead of CloudWatch Logs. This works because Kinesis natively supports cross account streaming, unlike CloudWatch Log Groups, so you can define the Kinesis stream in the logs account without deploying any additional infrastructure in the app accounts. You can then stream the log entries from Kinesis out to a CloudWatch Log Group in the logs account for searching and indexing.

• Use Elasticsearch as your log monitoring solution instead of CloudWatch Logs. In this approach, you would deploy a managed OpenSearch instance in the logs account that is available to each of the app environments (either using a public OpenSearch instance, or via VPC peering across accounts). You can then configure fluent-bit to ship the logs to the single OpenSearch instance in the logs account from each of the app accounts. From there, your developers can access the Kibana interface deployed with the OpenSearch interface to view the logs. You can stand up an OpenSearch instance using our service module.

[vj396]

for

You can then stream the log entries from Kinesis out to a CloudWatch Log Group in the logs account for searching and indexing.

Is there a recommended way, or a custom lambda to read from kinesis s3 bucket and ship logs to cloudwatch on every line write.

[yorinasub17]

Unfortunately, this is not something we have expertise in so we don't have an off the shelf recommendation for how to do this. We just know that this is a possible configuration that can be implemented.

That said, based on my understanding, using AWS Lambda is probably the best approach. This gives you the most flexibility, and you can even implement a custom filter in the function itself.