Skip to content

Commit 0aeaf26

Browse files
magreenbaummagreenbaum
authored
feat: Added support for Cloudwatch metric stream (#64)
Co-authored-by: magreenbaum <magreenbaum>
1 parent dd93152 commit 0aeaf26

File tree

16 files changed

+747
-0
lines changed

16 files changed

+747
-0
lines changed

‎README.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,50 @@ module "cis_alarms" {
120120

121121
AWS CloudTrail normally publishes logs into AWS CloudWatch Logs. This module creates log metric filters together with metric alarms according to [CIS AWS Foundations Benchmark v1.4.0 (05-28-2021)](https://www.cisecurity.org/benchmark/amazon_web_services/). Read more about [CIS AWS Foundations Controls](https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-cis-controls.html).
122122

123+
### Metric Stream
124+
125+
```hcl
126+
module "metric_stream" {
127+
name = "metric-stream"
128+
firehose_arn = "arn:aws:firehose:eu-west-1:835367859852:deliverystream/metric-stream-example"
129+
output_format = "json"
130+
role_arn = "arn:aws:iam::835367859852:role/metric-stream-to-firehose-20240113005123755300000002"
131+
132+
# conflicts with exclude_filter
133+
include_filter = {
134+
ec2 = {
135+
namespace = "AWS/EC2"
136+
metric_names = ["CPUUtilization", "NetworkIn"]
137+
}
138+
}
139+
140+
statistics_configuration = [
141+
{
142+
additional_statistics = ["p99"]
143+
include_metric = [
144+
{
145+
namespace = "AWS/EC2"
146+
metric_name = "CPUUtilization"
147+
},
148+
{
149+
namespace = "AWS/EC2"
150+
metric_name = "NetworkIn"
151+
}
152+
]
153+
},
154+
{
155+
additional_statistics = ["p90", "TM(10%:90%)"]
156+
include_metric = [
157+
{
158+
namespace = "AWS/EC2"
159+
metric_name = "CPUUtilization"
160+
}
161+
]
162+
}
163+
]
164+
}
165+
```
166+
123167
### Query Definition
124168

125169
```hcl
@@ -144,6 +188,7 @@ EOF
144188
- [Cloudwatch metric alarms for AWS Lambda with multiple dimensions](https://github.com/terraform-aws-modules/terraform-aws-cloudwatch/tree/master/examples/multiple-lambda-metric-alarm)
145189
- [CIS AWS Foundations Controls: Metrics + Alarms](https://github.com/terraform-aws-modules/terraform-aws-cloudwatch/tree/master/examples/cis-alarms)
146190
- [Cloudwatch query definition](https://github.com/terraform-aws-modules/terraform-aws-cloudwatch/tree/master/examples/query-definition)
191+
- [Cloudwatch Metric Stream](https://github.com/terraform-aws-modules/terraform-aws-cloudwatch/tree/master/examples/metric-stream)
147192

148193
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
149194
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

‎examples/metric-stream/README.md

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Cloudwatch Metric Stream
2+
3+
Configuration in this directory creates Cloudwatch metric streams and delivers them to Kinesis Firehose with an s3 destination.
4+
5+
## Usage
6+
7+
To run this example you need to execute:
8+
9+
```bash
10+
$ terraform init
11+
$ terraform plan
12+
$ terraform apply
13+
```
14+
15+
Note that this example may create resources which cost money. Run `terraform destroy` when you don't need these resources.
16+
17+
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
18+
## Requirements
19+
20+
| Name | Version |
21+
|------|---------|
22+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
23+
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.0 |
24+
| <a name="requirement_random"></a> [random](#requirement\_random) | >= 3.5 |
25+
26+
## Providers
27+
28+
| Name | Version |
29+
|------|---------|
30+
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.0 |
31+
| <a name="provider_random"></a> [random](#provider\_random) | >= 3.5 |
32+
33+
## Modules
34+
35+
| Name | Source | Version |
36+
|------|--------|---------|
37+
| <a name="module_firehose_to_s3"></a> [firehose\_to\_s3](#module\_firehose\_to\_s3) | terraform-aws-modules/iam/aws//modules/iam-assumable-role | >= 5.30 |
38+
| <a name="module_firehose_to_s3_policy"></a> [firehose\_to\_s3\_policy](#module\_firehose\_to\_s3\_policy) | terraform-aws-modules/iam/aws//modules/iam-policy | >= 5.30 |
39+
| <a name="module_metrics_bucket"></a> [metrics\_bucket](#module\_metrics\_bucket) | terraform-aws-modules/s3-bucket/aws | >= 3.15 |
40+
| <a name="module_stream_all"></a> [stream\_all](#module\_stream\_all) | ../../modules/metric-stream | n/a |
41+
| <a name="module_stream_all_disabled"></a> [stream\_all\_disabled](#module\_stream\_all\_disabled) | ../../modules/metric-stream | n/a |
42+
| <a name="module_stream_to_firehose_policy"></a> [stream\_to\_firehose\_policy](#module\_stream\_to\_firehose\_policy) | terraform-aws-modules/iam/aws//modules/iam-policy | >= 5.30 |
43+
| <a name="module_stream_to_firehose_role"></a> [stream\_to\_firehose\_role](#module\_stream\_to\_firehose\_role) | terraform-aws-modules/iam/aws//modules/iam-assumable-role | >= 5.30 |
44+
| <a name="module_stream_with_exclude_filter"></a> [stream\_with\_exclude\_filter](#module\_stream\_with\_exclude\_filter) | ../../modules/metric-stream | n/a |
45+
| <a name="module_stream_with_include_filter"></a> [stream\_with\_include\_filter](#module\_stream\_with\_include\_filter) | ../../modules/metric-stream | n/a |
46+
47+
## Resources
48+
49+
| Name | Type |
50+
|------|------|
51+
| [aws_kinesis_firehose_delivery_stream.s3_all_stream](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_firehose_delivery_stream) | resource |
52+
| [aws_kinesis_firehose_delivery_stream.s3_exclude_stream](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_firehose_delivery_stream) | resource |
53+
| [aws_kinesis_firehose_delivery_stream.s3_include_stream](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_firehose_delivery_stream) | resource |
54+
| [random_pet.this](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/pet) | resource |
55+
| [aws_iam_policy_document.firehose_to_s3](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
56+
| [aws_iam_policy_document.metric_stream_to_firehose](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
57+
58+
## Inputs
59+
60+
No inputs.
61+
62+
## Outputs
63+
64+
| Name | Description |
65+
|------|-------------|
66+
| <a name="output_all_example_metric_stream"></a> [all\_example\_metric\_stream](#output\_all\_example\_metric\_stream) | ARN of the metric stream. |
67+
| <a name="output_all_example_metric_stream_creation_date"></a> [all\_example\_metric\_stream\_creation\_date](#output\_all\_example\_metric\_stream\_creation\_date) | Date and time in RFC3339 format that the metric stream was created. |
68+
| <a name="output_all_example_metric_stream_last_update_date"></a> [all\_example\_metric\_stream\_last\_update\_date](#output\_all\_example\_metric\_stream\_last\_update\_date) | Date and time in RFC3339 format that the metric stream was last updated. |
69+
| <a name="output_all_example_metric_stream_state"></a> [all\_example\_metric\_stream\_state](#output\_all\_example\_metric\_stream\_state) | State of the metric stream. Possible values are running and stopped. |
70+
| <a name="output_exclude_example_metric_stream"></a> [exclude\_example\_metric\_stream](#output\_exclude\_example\_metric\_stream) | ARN of the metric stream. |
71+
| <a name="output_exclude_example_metric_stream_creation_date"></a> [exclude\_example\_metric\_stream\_creation\_date](#output\_exclude\_example\_metric\_stream\_creation\_date) | Date and time in RFC3339 format that the metric stream was created. |
72+
| <a name="output_exclude_example_metric_stream_last_update_date"></a> [exclude\_example\_metric\_stream\_last\_update\_date](#output\_exclude\_example\_metric\_stream\_last\_update\_date) | Date and time in RFC3339 format that the metric stream was last updated. |
73+
| <a name="output_exclude_example_metric_stream_state"></a> [exclude\_example\_metric\_stream\_state](#output\_exclude\_example\_metric\_stream\_state) | State of the metric stream. Possible values are running and stopped. |
74+
| <a name="output_include_example_metric_stream"></a> [include\_example\_metric\_stream](#output\_include\_example\_metric\_stream) | ARN of the metric stream. |
75+
| <a name="output_include_example_metric_stream_creation_date"></a> [include\_example\_metric\_stream\_creation\_date](#output\_include\_example\_metric\_stream\_creation\_date) | Date and time in RFC3339 format that the metric stream was created. |
76+
| <a name="output_include_example_metric_stream_last_update_date"></a> [include\_example\_metric\_stream\_last\_update\_date](#output\_include\_example\_metric\_stream\_last\_update\_date) | Date and time in RFC3339 format that the metric stream was last updated. |
77+
| <a name="output_include_example_metric_stream_state"></a> [include\_example\_metric\_stream\_state](#output\_include\_example\_metric\_stream\_state) | State of the metric stream. Possible values are running and stopped. |
78+
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

‎examples/metric-stream/main.tf

Lines changed: 234 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,234 @@
1+
provider "aws" {
2+
region = "eu-west-1"
3+
}
4+
5+
locals {
6+
name = "metric-stream"
7+
}
8+
9+
module "stream_with_exclude_filter" {
10+
source = "../../modules/metric-stream"
11+
12+
name = "${local.name}-with-exclude-filter"
13+
firehose_arn = aws_kinesis_firehose_delivery_stream.s3_exclude_stream.arn
14+
output_format = "json"
15+
role_arn = module.stream_to_firehose_role.iam_role_arn
16+
17+
# conflicts with include_filter
18+
exclude_filter = {
19+
logs = {
20+
namespace = "AWS/Logs"
21+
}
22+
usage = {
23+
namespace = "AWS/Usage"
24+
metric_names = ["CallCount", "ResourceCount"]
25+
}
26+
kms = {
27+
namespace = "AWS/Firehose"
28+
metric_names = ["KMSKeyDisabled"]
29+
}
30+
}
31+
}
32+
33+
module "stream_with_include_filter" {
34+
source = "../../modules/metric-stream"
35+
36+
name = "${local.name}-with-include-filter"
37+
firehose_arn = aws_kinesis_firehose_delivery_stream.s3_include_stream.arn
38+
output_format = "json"
39+
role_arn = module.stream_to_firehose_role.iam_role_arn
40+
41+
# conflicts with exclude_filter
42+
include_filter = {
43+
ec2 = {
44+
namespace = "AWS/EC2"
45+
metric_names = ["CPUUtilization", "NetworkIn"]
46+
}
47+
}
48+
49+
statistics_configuration = [
50+
{
51+
additional_statistics = ["p99"]
52+
include_metric = [
53+
{
54+
namespace = "AWS/EC2"
55+
metric_name = "CPUUtilization"
56+
},
57+
{
58+
namespace = "AWS/EC2"
59+
metric_name = "NetworkIn"
60+
}
61+
]
62+
},
63+
{
64+
additional_statistics = ["p90", "TM(10%:90%)"]
65+
include_metric = [
66+
{
67+
namespace = "AWS/EC2"
68+
metric_name = "CPUUtilization"
69+
}
70+
]
71+
}
72+
]
73+
}
74+
75+
module "stream_all" {
76+
source = "../../modules/metric-stream"
77+
78+
name = "${local.name}-all"
79+
firehose_arn = aws_kinesis_firehose_delivery_stream.s3_all_stream.arn
80+
output_format = "json"
81+
role_arn = module.stream_to_firehose_role.iam_role_arn
82+
}
83+
84+
module "stream_all_disabled" {
85+
source = "../../modules/metric-stream"
86+
87+
create = false
88+
89+
name = "${local.name}-all-disabled"
90+
firehose_arn = aws_kinesis_firehose_delivery_stream.s3_all_stream.arn
91+
output_format = "json"
92+
role_arn = module.stream_to_firehose_role.iam_role_arn
93+
}
94+
95+
resource "random_pet" "this" {
96+
length = 2
97+
}
98+
99+
module "metrics_bucket" {
100+
source = "terraform-aws-modules/s3-bucket/aws"
101+
version = ">= 3.15"
102+
103+
bucket = "${local.name}-${random_pet.this.id}"
104+
105+
force_destroy = true
106+
}
107+
108+
resource "aws_kinesis_firehose_delivery_stream" "s3_include_stream" {
109+
name = "${local.name}-include-filter-example"
110+
destination = "extended_s3"
111+
112+
extended_s3_configuration {
113+
role_arn = module.firehose_to_s3.iam_role_arn
114+
bucket_arn = module.metrics_bucket.s3_bucket_arn
115+
prefix = "include-filter/"
116+
}
117+
}
118+
119+
resource "aws_kinesis_firehose_delivery_stream" "s3_exclude_stream" {
120+
name = "${local.name}-exclude-filter-example"
121+
destination = "extended_s3"
122+
123+
extended_s3_configuration {
124+
role_arn = module.firehose_to_s3.iam_role_arn
125+
bucket_arn = module.metrics_bucket.s3_bucket_arn
126+
prefix = "exclude-filter/"
127+
}
128+
}
129+
130+
resource "aws_kinesis_firehose_delivery_stream" "s3_all_stream" {
131+
name = "${local.name}-all-example"
132+
destination = "extended_s3"
133+
134+
extended_s3_configuration {
135+
role_arn = module.firehose_to_s3.iam_role_arn
136+
bucket_arn = module.metrics_bucket.s3_bucket_arn
137+
prefix = "all/"
138+
}
139+
}
140+
141+
module "firehose_to_s3" {
142+
source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role"
143+
version = ">= 5.30"
144+
145+
trusted_role_services = [
146+
"firehose.amazonaws.com"
147+
]
148+
149+
create_role = true
150+
151+
role_name_prefix = "${local.name}-firehose-to-s3-"
152+
role_requires_mfa = false
153+
154+
custom_role_policy_arns = [
155+
module.firehose_to_s3_policy.arn
156+
]
157+
}
158+
159+
module "firehose_to_s3_policy" {
160+
source = "terraform-aws-modules/iam/aws//modules/iam-policy"
161+
version = ">= 5.30"
162+
163+
name = "${local.name}-firehose-to-s3"
164+
path = "/"
165+
description = local.name
166+
167+
policy = data.aws_iam_policy_document.firehose_to_s3.json
168+
}
169+
170+
data "aws_iam_policy_document" "firehose_to_s3" {
171+
statement {
172+
effect = "Allow"
173+
174+
actions = [
175+
"s3:AbortMultipartUpload",
176+
"s3:GetBucketLocation",
177+
"s3:GetObject",
178+
"s3:ListBucket",
179+
"s3:ListBucketMultipartUploads",
180+
"s3:PutObject",
181+
]
182+
183+
resources = [
184+
module.metrics_bucket.s3_bucket_arn,
185+
"${module.metrics_bucket.s3_bucket_arn}/*",
186+
]
187+
}
188+
}
189+
190+
module "stream_to_firehose_role" {
191+
source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role"
192+
version = ">= 5.30"
193+
194+
trusted_role_services = [
195+
"streams.metrics.cloudwatch.amazonaws.com"
196+
]
197+
198+
create_role = true
199+
200+
role_name_prefix = "${local.name}-to-firehose-"
201+
role_requires_mfa = false
202+
203+
custom_role_policy_arns = [
204+
module.stream_to_firehose_policy.arn
205+
]
206+
}
207+
208+
module "stream_to_firehose_policy" {
209+
source = "terraform-aws-modules/iam/aws//modules/iam-policy"
210+
version = ">= 5.30"
211+
212+
name = "${local.name}-to-firehose"
213+
path = "/"
214+
description = local.name
215+
216+
policy = data.aws_iam_policy_document.metric_stream_to_firehose.json
217+
}
218+
219+
data "aws_iam_policy_document" "metric_stream_to_firehose" {
220+
statement {
221+
effect = "Allow"
222+
223+
actions = [
224+
"firehose:PutRecord",
225+
"firehose:PutRecordBatch",
226+
]
227+
228+
resources = [
229+
aws_kinesis_firehose_delivery_stream.s3_include_stream.arn,
230+
aws_kinesis_firehose_delivery_stream.s3_exclude_stream.arn,
231+
aws_kinesis_firehose_delivery_stream.s3_all_stream.arn,
232+
]
233+
}
234+
}

0 commit comments

Comments
 (0)