[FEAT] Add Terraform configuration for AWS with Backend Private Link#24
[FEAT] Add Terraform configuration for AWS with Backend Private Link#24eri-adepoju wants to merge 2 commits intomainfrom
Conversation
| # ============================================================================= | ||
|
|
||
| # Egress ports for Databricks connectivity | ||
| sg_egress_ports = [443, 3306, 2443, 8443, 8444, 8445, 8446, 8447, 8448, 8449, 8450, 8451] |
There was a problem hiding this comment.
Let's add 6666 to this list as well! By default it is required for SCC.
| cidr_blocks = [egress.value] | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
We're missing an egress rule to S3. We should add the following lines
dynamic "egress" { for_each = var.network_configuration != "custom" ? [1] : [] content { description = "S3 Gateway Endpoint - SG" from_port = 443 to_port = 443 protocol = "tcp" prefix_list_ids = [data.aws_prefix_list.s3.id] } }
SRA does the same. check network.tf (Lines 76-85)
I could not get the clusters to spin up without adding this rule, or without allowing 0.0.0.0/0 outbound.
| endpoints = { | ||
| s3 = { | ||
| service = "s3" | ||
| service_type = "Gateway" | ||
| route_table_ids = length(var.private_route_table_ids) > 0 ? var.private_route_table_ids : module.vpc[0].private_route_table_ids | ||
| }, |
There was a problem hiding this comment.
We should think about what happens when a customer already has an s3 gateway endpoint in his VPC (very common)
| enable_dns_hostnames = true | ||
| enable_nat_gateway = true | ||
| single_nat_gateway = true | ||
| one_nat_gateway_per_az = false | ||
| create_igw = true |
There was a problem hiding this comment.
We should make it optional to deploy the VPC with a NAT Gateway. We can get a variable to assign these values to true or false.
| public_subnet_names = [for az in var.availability_zones : format("%s-public-%s", var.resource_prefix, az)] | ||
| public_subnets = var.public_subnets_cidr |
There was a problem hiding this comment.
Same applies to public subnets - only create them if specified.
|
In addition to the comments I shared, we should think about the risks of creating resources in the VPCs that customers can optionally provide. SRA, for example, either creates all the objects or assumes the customer's VPC already has the correct configs. |
|
This is a minor piece of feedback, but could you add a description of this new scenario to the README.md at the link below? |
|
Could we change the project name to aws-byovpc-classic-privatelink? |
3 participants
Pull Request
Description
Adds.a new AWS workspace deployment template with backend private link.
Category
Type of Change
Project Details
Project Name: AWS Databricks workspace with backend private link deployment
Purpose: Demonstrate how to deploy Databricks workspaces with a new or existing VPC and backend private link.
Technologies Used: Terraform
Testing
Security Compliance ✅
By submitting this PR, I confirm I have followed the CONTRIBUTING.md guidelines and security requirements.