[FEAT] Add Terraform configuration for AWS with Backend Private Link

workspace-setup/terraform-examples/README.md

@@ -70,6 +70,7 @@ This naming helps you quickly identify the right scenario for your needs.
 | Scenario | Description |
 |----------|-------------|
 | [aws-byovpc](./aws/aws-byovpc/) | Deploy a Databricks workspace using "Bring Your Own VPC" (BYOVPC) pattern with Unity Catalog Metastore. Create a new VPC or use an existing one with full control over network infrastructure. |
+| [aws-byovpc-classic-privatelink](./aws/aws-byovpc-classic-privatelink/) | Deploy a Databricks workspace with classic Private Link (REST API and SCC relay). Choose **standard** (template creates VPC with NAT/IGW and S3/STS/Kinesis endpoints), **fully_private** (no NAT/IGW, dedicated endpoint subnet), or **custom** (you supply VPC, subnets, security groups, and backend VPC endpoint IDs; no AWS networking created). Optional Unity Catalog metastore creation or attachment. |
 
 ### Azure
 | Scenario | Description |

workspace-setup/terraform-examples/aws/aws-byovpc-classic-privatelink/.gitignore

@@ -0,0 +1,42 @@
+# Local .terraform directories
+*/.terraform/*
+*/.terraform
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+*.tfstate.backup
+
+# Crash log files
+crash.log
+crash.*.log
+
+# Ignore CLI configuration files
+.terraformrc
+terraform.rc
+
+# Ignore .tfvars files that contain secrets
+# Keep .tfvars.example as template
+terraform.tfvars
+*auto.tfvars
+*.auto.tfvars
+
+# Ignore override files as they are usually used to override resources locally and so
+# are not checked in
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json
+
+# Include override files you do wish to add to version control using negated pattern
+#
+# !example_override.tf
+
+# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
+# example: *tfplan*
+
+# MAC Control
+.DS_Store
+
+# IntelliJ
+.idea/

workspace-setup/terraform-examples/aws/aws-byovpc-classic-privatelink/README.md

@@ -0,0 +1,211 @@
+# Databricks on AWS with Private Link (BYOVPC)
+
+This Terraform configuration deploys a Databricks workspace on AWS using the "Bring Your Own VPC" (BYOVPC) pattern with **Private Link** for backend REST API and Secure Cluster Connectivity (SCC) relay. Traffic between your VPC and Databricks stays on the AWS network.
+
+## Architecture Overview
+
+This configuration creates:
+- **Databricks Workspace**: Enterprise-tier workspace with Private Access Settings
+- **Private Link**: VPC endpoints for Databricks REST API and SCC relay (pre-configured per region)
+- **VPC Infrastructure**: Customer-managed VPC with public/private subnets (or use existing VPC)
+- **Security**: IAM roles, workspace and Private Link security groups, S3 bucket policies
+- **Unity Catalog**: Optional metastore creation or attachment to existing metastore
+- **Additional VPC Endpoints**: STS and Kinesis (for Databricks services over Private Link)
+
+## Prerequisites
+
+1. **Terraform**: Version 1.3 or higher
+2. **AWS Account**: With appropriate permissions to create VPC, IAM, S3, and VPC endpoint resources
+3. **Databricks Account**: E2 account with account admin access
+4. **AWS CLI**: Configured with valid credentials
+5. **Databricks CLI**: Service principal credentials for account-level operations
+
+## Required Permissions
+
+### AWS Permissions
+- VPC, Subnet, Internet Gateway, NAT Gateway, Route Table management
+- IAM Role and Policy management
+- S3 Bucket creation and policy management
+- Security Group management
+- VPC Endpoint (Interface) creation and management
+
+### Databricks Permissions
+- Account Admin access to create workspaces and Private Access Settings
+- Ability to create credentials, storage configs, networks, and VPC endpoint configs
+
+## Quick Start
+
+### 1. Configure Authentication
+
+#### AWS Authentication
+```bash
+# Option 1: Environment variables
+export AWS_ACCESS_KEY_ID="your-access-key"
+export AWS_SECRET_ACCESS_KEY="your-secret-key"
+export AWS_SESSION_TOKEN="your-session-token"  # If using temporary credentials
+
+# Option 2: AWS CLI profile
+export AWS_PROFILE="your-profile"
+```
+
+#### Databricks Authentication
+```bash
+# Use service principal for account-level operations
+export DATABRICKS_CLIENT_ID="your-client-id"
+export DATABRICKS_CLIENT_SECRET="your-client-secret"
+```
+
+### 2. Configure Variables
+
+```bash
+cd tf/
+cp terraform.tfvars.example terraform.tfvars
+```
+
+Edit `terraform.tfvars` with your specific values:
+```hcl
+databricks_account_id = "your-account-id"
+prefix                = "my-workspace"
+resource_prefix       = "my-workspace"
+region                = "us-west-2"
+
+# Network: "standard" or "fully_private" (template creates VPC) or "custom" (you supply IDs)
+network_configuration = "standard"
+vpc_cidr_range        = "10.0.0.0/16"
+availability_zones    = ["us-west-2a", "us-west-2b"]
+private_subnets_cidr  = ["10.0.1.0/24", "10.0.2.0/24"]
+public_subnets_cidr   = ["10.0.101.0/24", "10.0.102.0/24"]
+
+# Unity Catalog
+metastore_name = "my-metastore"
+```
+
+### 3. Deploy
+
+```bash
+# Initialize Terraform
+terraform init
+
+# Review the execution plan
+terraform plan
+
+# Apply the configuration
+terraform apply
+```
+
+The deployment typically takes 10-15 minutes.
+
+### 4. Access Your Workspace
+
+After successful deployment:
+```bash
+# Get the workspace URL
+terraform output workspace_url
+
+# Get the workspace ID
+terraform output workspace_id
+```
+
+Navigate to the workspace URL and log in with your Databricks credentials.
+
+## Configuration Options
+
+### Network configuration
+
+Set `network_configuration` to choose how networking is managed:
+
+- **Pathway 1 – Template-owned:** `"standard"` or `"fully_private"`. The template creates the VPC and all networking (security groups, VPC endpoints). Leave `vpc_id` empty.
+- **Pathway 2 – Custom:** `"custom"`. You supply `vpc_id`, `subnet_ids`, `security_group_ids`, and the two AWS VPC endpoint IDs (backend REST and SCC relay). The template creates no AWS networking; it only registers your endpoints with Databricks and creates the workspace.
+
+See [Step 5: Add VPC endpoints for other AWS services](https://docs.databricks.com/aws/en/security/network/classic/privatelink#step-5-add-vpc-endpoints-for-other-aws-services) (Option 1 = standard, Option 2 = fully private) and the [SRA AWS template](https://github.com/databricks/terraform-databricks-sra/tree/main/aws) for reference.
+
+### Pathway 1 – Template-owned (standard or fully_private)
+
+Set `network_configuration = "standard"` or `"fully_private"` and configure:
+
+```hcl
+network_configuration = "standard"   # or "fully_private"
+vpc_cidr_range        = "10.0.0.0/16"
+availability_zones    = ["us-west-2a", "us-west-2b"]
+private_subnets_cidr  = ["10.0.1.0/24", "10.0.2.0/24"]
+public_subnets_cidr   = ["10.0.101.0/24", "10.0.102.0/24"]
+```
+
+- **standard:** VPC with NAT Gateway and Internet Gateway; S3 gateway, STS, and Kinesis VPC endpoints in workspace subnets. Use when compute can use the internet for optional traffic.
+- **fully_private:** VPC with no NAT/IGW; dedicated subnet for AWS service endpoints (min /27) and S3/STS/Kinesis endpoints. Use for strict no-internet (air-gap) deployments. Ensure `endpoint_subnet_cidr` (default `10.0.3.0/27`) does not overlap `private_subnets_cidr`.
+
+### Pathway 2 – Custom
+
+Set `network_configuration = "custom"` and supply IDs for existing resources. The template creates no VPC, subnets, security groups, or VPC endpoints.
+
+```hcl
+network_configuration     = "custom"
+vpc_id                    = "vpc-xxxxxxxxx"
+subnet_ids                = ["subnet-xxxxxxxxx", "subnet-yyyyyyyyy"]
+security_group_ids        = ["sg-xxxxxxxxx"]
+backend_rest_aws_vpce_id   = "vpce-xxxxxxxxx"
+backend_relay_aws_vpce_id  = "vpce-yyyyyyyyy"
+```
+
+**Creating network resources first (Pathway 2):** Create the VPC, subnets, workspace and Private Link security groups, and the two Databricks Private Link VPC endpoints (workspace REST + SCC relay) in AWS before applying. Use these docs:
+
+- [Configure a customer-managed VPC](https://docs.databricks.com/aws/en/security/network/classic/customer-managed-vpc) – VPC settings, CIDR, subnets
+- [Configure classic private connectivity – Step 1 – Create security groups](https://docs.databricks.com/aws/en/security/network/classic/privatelink#step-1-configure-aws-network-objects) – Workspace SG and Private Link endpoint SG
+- [Step 2: Create VPC endpoints](https://docs.databricks.com/aws/en/security/network/classic/privatelink#step-2-create-vpc-endpoints) – Workspace and SCC relay interface endpoints; the resulting AWS VPC endpoint IDs are what you pass as `backend_rest_aws_vpce_id` and `backend_relay_aws_vpce_id`
+- [Step 5: Add VPC endpoints for other AWS services](https://docs.databricks.com/aws/en/security/network/classic/privatelink#step-5-add-vpc-endpoints-for-other-aws-services) – Optional S3/STS/Kinesis if clusters need them
+- [AWS VPC endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints.html) – General reference
+
+Requirements: at least 2 private subnets in different AZs; security group egress allowing required Databricks ports.
+
+### Workspace Security Group Egress: Restrictive vs Permissive
+
+- **Restrictive (default):** Egress for `sg_egress_ports` is limited to `var.vpc_cidr_range` and optionally `var.additional_egress_ips`. Use for least-privilege with Private Link.
+- **Permissive:** Same ports allowed to `0.0.0.0/0`. A permissive block is commented out in `network.tf`; comment out the restrictive egress block and uncomment the one with `cidr_blocks = ["0.0.0.0/0"]` to switch.
+
+### Unity Catalog Options
+
+#### Create New Metastore
+```hcl
+metastore_id   = ""
+metastore_name = "my-metastore"
+```
+
+#### Use Existing Metastore
+```hcl
+metastore_id   = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
+metastore_name = ""  # Not required when using existing
+```
+
+## File Structure
+
+This project uses a flat, organized structure with purpose-specific files:
+
+```
+tf/
+├── versions.tf              # Terraform and provider version constraints
+├── providers.tf             # Provider configurations (AWS, Databricks)
+├── variables.tf             # All input variable definitions
+├── locals.tf                # Endpoint ID locals (template-created vs user-provided)
+├── outputs.tf               # All output values
+├── terraform.tfvars.example  # Configuration template
+├── credential.tf            # IAM cross-account role and policies
+├── network.tf               # VPC (template-owned), workspace SG, endpoint subnet (fully_private)
+├── privatelink.tf            # Private Link SG, AWS endpoints SG (fully_private), backend VPC endpoints
+├── endpoints.tf             # S3/STS/Kinesis VPC endpoints (template-owned only)
+├── root_bucket.tf           # S3 bucket for workspace root storage
+├── workspace.tf             # Databricks workspace, MWS networks, PAS, VPC endpoint configs
+└── metastore.tf             # Unity Catalog metastore
+```
+
+Terraform loads all `.tf` files in the directory; the structure is organizational only.
+
+## Documentation
+
+Please note that the code in this template is provided for your exploration only and is not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS, and we do not make any guarantees of any kind.
+
+- [Deploy with customer-managed VPC](https://docs.databricks.com/aws/en/security/network/classic/customer-managed-vpc)
+- [Configure a Databricks workspace with Private Link](https://docs.databricks.com/aws/en/security/network/classic/privatelink)
+- [Security Reference Architecture Template](https://github.com/databricks/terraform-databricks-sra/tree/main/aws)
+    - This is a template that adheres to the best security practices we recommend.
+- [Terraform Databricks provider documentation](https://registry.terraform.io/providers/databricks/databricks/latest/docs)
+

workspace-setup/terraform-examples/aws/aws-byovpc-classic-privatelink/tf/credential.tf

@@ -0,0 +1,21 @@
+data "databricks_aws_assume_role_policy" "this" {
+  provider    = databricks.mws
+  external_id = var.databricks_account_id
+}
+
+resource "aws_iam_role" "cross_account_role" {
+  name               = "${var.prefix}-crossaccount"
+  assume_role_policy = data.databricks_aws_assume_role_policy.this.json
+  tags               = var.tags
+}
+
+data "databricks_aws_crossaccount_policy" "this" {
+  provider    = databricks.mws
+  policy_type = "customer"
+}
+
+resource "aws_iam_role_policy" "this" {
+  name   = "${var.prefix}-policy"
+  role   = aws_iam_role.cross_account_role.id
+  policy = data.databricks_aws_crossaccount_policy.this.json
+}