Skip to content

Commit 8cf1088

Browse files
jbeemsterjbeemster
committed
Update aws/pipeline to support AWS China (closes #95)
1 parent 45429b3 commit 8cf1088

20 files changed

+132
-56
lines changed

‎terraform/aws/pipeline/default/README.md

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -18,25 +18,25 @@
1818
|------|--------|---------|
1919
| <a name="module_bad_1_stream"></a> [bad\_1\_stream](#module\_bad\_1\_stream) | snowplow-devops/kinesis-stream/aws | 0.3.0 |
2020
| <a name="module_bad_2_stream"></a> [bad\_2\_stream](#module\_bad\_2\_stream) | snowplow-devops/kinesis-stream/aws | 0.3.0 |
21-
| <a name="module_collector_kinesis"></a> [collector\_kinesis](#module\_collector\_kinesis) | snowplow-devops/collector-kinesis-ec2/aws | 0.9.0 |
21+
| <a name="module_collector_kinesis"></a> [collector\_kinesis](#module\_collector\_kinesis) | snowplow-devops/collector-kinesis-ec2/aws | 0.9.1 |
2222
| <a name="module_collector_lb"></a> [collector\_lb](#module\_collector\_lb) | snowplow-devops/alb/aws | 0.2.0 |
23-
| <a name="module_db_loader"></a> [db\_loader](#module\_db\_loader) | snowplow-devops/databricks-loader-ec2/aws | 0.2.0 |
24-
| <a name="module_db_transformer_wrp"></a> [db\_transformer\_wrp](#module\_db\_transformer\_wrp) | snowplow-devops/transformer-kinesis-ec2/aws | 0.4.0 |
25-
| <a name="module_enrich_kinesis"></a> [enrich\_kinesis](#module\_enrich\_kinesis) | snowplow-devops/enrich-kinesis-ec2/aws | 0.6.0 |
23+
| <a name="module_db_loader"></a> [db\_loader](#module\_db\_loader) | snowplow-devops/databricks-loader-ec2/aws | 0.2.1 |
24+
| <a name="module_db_transformer_wrp"></a> [db\_transformer\_wrp](#module\_db\_transformer\_wrp) | snowplow-devops/transformer-kinesis-ec2/aws | 0.4.1 |
25+
| <a name="module_enrich_kinesis"></a> [enrich\_kinesis](#module\_enrich\_kinesis) | snowplow-devops/enrich-kinesis-ec2/aws | 0.6.1 |
2626
| <a name="module_enriched_stream"></a> [enriched\_stream](#module\_enriched\_stream) | snowplow-devops/kinesis-stream/aws | 0.3.0 |
27-
| <a name="module_postgres_loader_bad"></a> [postgres\_loader\_bad](#module\_postgres\_loader\_bad) | snowplow-devops/postgres-loader-kinesis-ec2/aws | 0.5.0 |
28-
| <a name="module_postgres_loader_enriched"></a> [postgres\_loader\_enriched](#module\_postgres\_loader\_enriched) | snowplow-devops/postgres-loader-kinesis-ec2/aws | 0.5.0 |
27+
| <a name="module_postgres_loader_bad"></a> [postgres\_loader\_bad](#module\_postgres\_loader\_bad) | snowplow-devops/postgres-loader-kinesis-ec2/aws | 0.5.1 |
28+
| <a name="module_postgres_loader_enriched"></a> [postgres\_loader\_enriched](#module\_postgres\_loader\_enriched) | snowplow-devops/postgres-loader-kinesis-ec2/aws | 0.5.1 |
2929
| <a name="module_postgres_loader_rds"></a> [postgres\_loader\_rds](#module\_postgres\_loader\_rds) | snowplow-devops/rds/aws | 0.4.0 |
3030
| <a name="module_raw_stream"></a> [raw\_stream](#module\_raw\_stream) | snowplow-devops/kinesis-stream/aws | 0.3.0 |
31-
| <a name="module_rs_loader"></a> [rs\_loader](#module\_rs\_loader) | snowplow-devops/redshift-loader-ec2/aws | 0.2.0 |
32-
| <a name="module_rs_transformer_stsv"></a> [rs\_transformer\_stsv](#module\_rs\_transformer\_stsv) | snowplow-devops/transformer-kinesis-ec2/aws | 0.4.0 |
33-
| <a name="module_s3_loader_bad"></a> [s3\_loader\_bad](#module\_s3\_loader\_bad) | snowplow-devops/s3-loader-kinesis-ec2/aws | 0.5.0 |
34-
| <a name="module_s3_loader_enriched"></a> [s3\_loader\_enriched](#module\_s3\_loader\_enriched) | snowplow-devops/s3-loader-kinesis-ec2/aws | 0.5.0 |
35-
| <a name="module_s3_loader_raw"></a> [s3\_loader\_raw](#module\_s3\_loader\_raw) | snowplow-devops/s3-loader-kinesis-ec2/aws | 0.5.0 |
31+
| <a name="module_rs_loader"></a> [rs\_loader](#module\_rs\_loader) | snowplow-devops/redshift-loader-ec2/aws | 0.2.1 |
32+
| <a name="module_rs_transformer_stsv"></a> [rs\_transformer\_stsv](#module\_rs\_transformer\_stsv) | snowplow-devops/transformer-kinesis-ec2/aws | 0.4.1 |
33+
| <a name="module_s3_loader_bad"></a> [s3\_loader\_bad](#module\_s3\_loader\_bad) | snowplow-devops/s3-loader-kinesis-ec2/aws | 0.5.1 |
34+
| <a name="module_s3_loader_enriched"></a> [s3\_loader\_enriched](#module\_s3\_loader\_enriched) | snowplow-devops/s3-loader-kinesis-ec2/aws | 0.5.1 |
35+
| <a name="module_s3_loader_raw"></a> [s3\_loader\_raw](#module\_s3\_loader\_raw) | snowplow-devops/s3-loader-kinesis-ec2/aws | 0.5.1 |
3636
| <a name="module_s3_pipeline_bucket"></a> [s3\_pipeline\_bucket](#module\_s3\_pipeline\_bucket) | snowplow-devops/s3-bucket/aws | 0.2.0 |
37-
| <a name="module_sf_loader"></a> [sf\_loader](#module\_sf\_loader) | snowplow-devops/snowflake-loader-ec2/aws | 0.3.0 |
38-
| <a name="module_sf_transformer_wrj"></a> [sf\_transformer\_wrj](#module\_sf\_transformer\_wrj) | snowplow-devops/transformer-kinesis-ec2/aws | 0.4.0 |
39-
| <a name="module_snowflake_streaming_loader_enriched"></a> [snowflake\_streaming\_loader\_enriched](#module\_snowflake\_streaming\_loader\_enriched) | snowplow-devops/snowflake-streaming-loader-ec2/aws | 0.1.0 |
37+
| <a name="module_sf_loader"></a> [sf\_loader](#module\_sf\_loader) | snowplow-devops/snowflake-loader-ec2/aws | 0.3.1 |
38+
| <a name="module_sf_transformer_wrj"></a> [sf\_transformer\_wrj](#module\_sf\_transformer\_wrj) | snowplow-devops/transformer-kinesis-ec2/aws | 0.4.1 |
39+
| <a name="module_snowflake_streaming_loader_enriched"></a> [snowflake\_streaming\_loader\_enriched](#module\_snowflake\_streaming\_loader\_enriched) | snowplow-devops/snowflake-streaming-loader-ec2/aws | 0.1.2 |
4040

4141
## Resources
4242

@@ -79,6 +79,7 @@
7979
| <a name="input_postgres_db_password"></a> [postgres\_db\_password](#input\_postgres\_db\_password) | The password to use to connect to the database | `string` | `""` | no |
8080
| <a name="input_postgres_db_publicly_accessible"></a> [postgres\_db\_publicly\_accessible](#input\_postgres\_db\_publicly\_accessible) | Whether to make the Postgres RDS instance accessible over the internet | `bool` | `false` | no |
8181
| <a name="input_postgres_db_username"></a> [postgres\_db\_username](#input\_postgres\_db\_username) | The username to use to connect to the database | `string` | `""` | no |
82+
| <a name="input_private_ecr_registry"></a> [private\_ecr\_registry](#input\_private\_ecr\_registry) | The URL of an ECR registry that the sub-account has access to (e.g. '000000000000.dkr.ecr.cn-north-1.amazonaws.com.cn/') | `string` | `""` | no |
8283
| <a name="input_redshift_database"></a> [redshift\_database](#input\_redshift\_database) | Redshift database name | `string` | `""` | no |
8384
| <a name="input_redshift_enabled"></a> [redshift\_enabled](#input\_redshift\_enabled) | Whether to enable loading into a Redshift Database | `bool` | `false` | no |
8485
| <a name="input_redshift_host"></a> [redshift\_host](#input\_redshift\_host) | Redshift cluster hostname | `string` | `""` | no |

‎terraform/aws/pipeline/default/main.tf

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ module "collector_lb" {
8686

8787
module "collector_kinesis" {
8888
source = "snowplow-devops/collector-kinesis-ec2/aws"
89-
version = "0.9.0"
89+
version = "0.9.1"
9090

9191
accept_limited_use_license = var.accept_limited_use_license
9292

@@ -113,12 +113,14 @@ module "collector_kinesis" {
113113

114114
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
115115
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
116+
117+
private_ecr_registry = var.private_ecr_registry
116118
}
117119

118120
# 3. Deploy Enrichment
119121
module "enrich_kinesis" {
120122
source = "snowplow-devops/enrich-kinesis-ec2/aws"
121-
version = "0.6.0"
123+
version = "0.6.1"
122124

123125
accept_limited_use_license = var.accept_limited_use_license
124126

@@ -148,4 +150,6 @@ module "enrich_kinesis" {
148150

149151
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
150152
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
153+
154+
private_ecr_registry = var.private_ecr_registry
151155
}

‎terraform/aws/pipeline/default/target_amazon_s3.tf

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
module "s3_loader_raw" {
22
source = "snowplow-devops/s3-loader-kinesis-ec2/aws"
3-
version = "0.5.0"
3+
version = "0.5.1"
44

55
accept_limited_use_license = var.accept_limited_use_license
66

@@ -28,11 +28,13 @@ module "s3_loader_raw" {
2828

2929
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
3030
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
31+
32+
private_ecr_registry = var.private_ecr_registry
3133
}
3234

3335
module "s3_loader_bad" {
3436
source = "snowplow-devops/s3-loader-kinesis-ec2/aws"
35-
version = "0.5.0"
37+
version = "0.5.1"
3638

3739
accept_limited_use_license = var.accept_limited_use_license
3840

@@ -63,11 +65,13 @@ module "s3_loader_bad" {
6365

6466
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
6567
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
68+
69+
private_ecr_registry = var.private_ecr_registry
6670
}
6771

6872
module "s3_loader_enriched" {
6973
source = "snowplow-devops/s3-loader-kinesis-ec2/aws"
70-
version = "0.5.0"
74+
version = "0.5.1"
7175

7276
accept_limited_use_license = var.accept_limited_use_license
7377

@@ -97,4 +101,6 @@ module "s3_loader_enriched" {
97101

98102
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
99103
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
104+
105+
private_ecr_registry = var.private_ecr_registry
100106
}

‎terraform/aws/pipeline/default/target_databricks.tf

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ resource "aws_sqs_queue" "db_message_queue" {
99

1010
module "db_transformer_wrp" {
1111
source = "snowplow-devops/transformer-kinesis-ec2/aws"
12-
version = "0.4.0"
12+
version = "0.4.1"
1313

1414
accept_limited_use_license = var.accept_limited_use_license
1515

@@ -44,11 +44,13 @@ module "db_transformer_wrp" {
4444

4545
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
4646
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
47+
48+
private_ecr_registry = var.private_ecr_registry
4749
}
4850

4951
module "db_loader" {
5052
source = "snowplow-devops/databricks-loader-ec2/aws"
51-
version = "0.2.0"
53+
version = "0.2.1"
5254

5355
accept_limited_use_license = var.accept_limited_use_license
5456

@@ -82,4 +84,6 @@ module "db_loader" {
8284

8385
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
8486
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
87+
88+
private_ecr_registry = var.private_ecr_registry
8589
}

‎terraform/aws/pipeline/default/target_postgres.tf

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ module "postgres_loader_rds" {
2121

2222
module "postgres_loader_enriched" {
2323
source = "snowplow-devops/postgres-loader-kinesis-ec2/aws"
24-
version = "0.5.0"
24+
version = "0.5.1"
2525

2626
accept_limited_use_license = var.accept_limited_use_license
2727

@@ -59,11 +59,13 @@ module "postgres_loader_enriched" {
5959

6060
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
6161
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
62+
63+
private_ecr_registry = var.private_ecr_registry
6264
}
6365

6466
module "postgres_loader_bad" {
6567
source = "snowplow-devops/postgres-loader-kinesis-ec2/aws"
66-
version = "0.5.0"
68+
version = "0.5.1"
6769

6870
accept_limited_use_license = var.accept_limited_use_license
6971

@@ -101,4 +103,6 @@ module "postgres_loader_bad" {
101103

102104
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
103105
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
106+
107+
private_ecr_registry = var.private_ecr_registry
104108
}

‎terraform/aws/pipeline/default/target_redshift.tf

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ resource "aws_sqs_queue" "rs_message_queue" {
99

1010
module "rs_transformer_stsv" {
1111
source = "snowplow-devops/transformer-kinesis-ec2/aws"
12-
version = "0.4.0"
12+
version = "0.4.1"
1313

1414
accept_limited_use_license = var.accept_limited_use_license
1515

@@ -44,11 +44,13 @@ module "rs_transformer_stsv" {
4444

4545
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
4646
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
47+
48+
private_ecr_registry = var.private_ecr_registry
4749
}
4850

4951
module "rs_loader" {
5052
source = "snowplow-devops/redshift-loader-ec2/aws"
51-
version = "0.2.0"
53+
version = "0.2.1"
5254

5355
accept_limited_use_license = var.accept_limited_use_license
5456

@@ -82,4 +84,6 @@ module "rs_loader" {
8284

8385
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
8486
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
87+
88+
private_ecr_registry = var.private_ecr_registry
8589
}

‎terraform/aws/pipeline/default/target_snowflake.tf

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ resource "aws_sqs_queue" "sf_message_queue" {
99

1010
module "sf_transformer_wrj" {
1111
source = "snowplow-devops/transformer-kinesis-ec2/aws"
12-
version = "0.4.0"
12+
version = "0.4.1"
1313

1414
accept_limited_use_license = var.accept_limited_use_license
1515

@@ -44,11 +44,13 @@ module "sf_transformer_wrj" {
4444

4545
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
4646
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
47+
48+
private_ecr_registry = var.private_ecr_registry
4749
}
4850

4951
module "sf_loader" {
5052
source = "snowplow-devops/snowflake-loader-ec2/aws"
51-
version = "0.3.0"
53+
version = "0.3.1"
5254

5355
accept_limited_use_license = var.accept_limited_use_license
5456

@@ -83,4 +85,6 @@ module "sf_loader" {
8385

8486
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
8587
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
88+
89+
private_ecr_registry = var.private_ecr_registry
8690
}

‎terraform/aws/pipeline/default/target_snowflake_streaming.tf

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
module "snowflake_streaming_loader_enriched" {
22
source = "snowplow-devops/snowflake-streaming-loader-ec2/aws"
3-
version = "0.1.0"
3+
version = "0.1.2"
44

55
accept_limited_use_license = var.accept_limited_use_license
66

@@ -33,4 +33,6 @@ module "snowflake_streaming_loader_enriched" {
3333

3434
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
3535
cloudwatch_logs_retention_days = var.cloudwatch_logs_retention_days
36+
37+
private_ecr_registry = var.private_ecr_registry
3638
}

‎terraform/aws/pipeline/default/terraform.tfvars

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,3 +129,6 @@ cloudwatch_logs_retention_days = 7
129129

130130
# Extra Tags to append to created resources (optional)
131131
tags = {}
132+
133+
# Image Repositories
134+
private_ecr_registry = ""

‎terraform/aws/pipeline/default/variables.tf

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,12 @@ variable "cloudwatch_logs_retention_days" {
121121
type = number
122122
}
123123

124+
variable "private_ecr_registry" {
125+
description = "The URL of an ECR registry that the sub-account has access to (e.g. '000000000000.dkr.ecr.cn-north-1.amazonaws.com.cn/')"
126+
type = string
127+
default = ""
128+
}
129+
124130
# --- Target: Amazon S3
125131

126132
variable "s3_raw_enabled" {

0 commit comments

Comments
 (0)