feat: add --switch-config-path parameter to llm-transpile command

[hiroyukinakazato-db]

Summary

• Add optional --switch-config-path parameter to llm-transpile CLI command
• Parameter allows users to specify a custom Switch configuration file in the workspace
• Path must start with /Workspace/ (validation included)
• When not specified, Switch uses its default configuration

Closes #2255

Test plan

• Unit tests added and passing
• E2E test: verify --switch-config-path works with actual workspace execution
• Verify Switch side correctly reads switch_config_path job parameter

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.43%. Comparing base (7ff77c2) to head (7cbc358).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2256      +/-   ##
==========================================
+ Coverage   66.41%   66.43%   +0.02%     
==========================================
  Files          99       99              
  Lines        9094     9100       +6     
  Branches      974      977       +3     
==========================================
+ Hits         6040     6046       +6     
  Misses       2878     2878              
  Partials      176      176              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ 145/145 passed, 8 flaky, 4 skipped, 47m12s total

Flaky tests:

• 🤪 test_installs_and_runs_local_bladebridge (22.013s)
• 🤪 test_installs_and_runs_pypi_bladebridge (27.97s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (18.636s)
• 🤪 test_transpiles_informatica_to_sparksql (18.745s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (18.558s)
• 🤪 test_transpile_teradata_sql (21.628s)
• 🤪 test_transpile_teradata_sql_non_interactive[True] (6.077s)
• 🤪 test_transpile_teradata_sql_non_interactive[False] (5.796s)

_{Running from acceptance #4014}

[hiroyukinakazato-db]

E2E Test Evidence

Test 1: Without --switch-config-path (default behavior)

Command:

databricks labs lakebridge llm-transpile \
  --input-source /tmp/switch_e2e_test \
  --output-ws-folder /Workspace/Users/hiroyuki.nakazato@databricks.com/switch-io/output/test-no-config \
  --source-dialect mssql \
  --catalog-name hinak_catalog_aws_apne1 \
  --schema-name switch \
  --volume switch_volume \
  --foundation-model databricks-claude-sonnet-4-5 \
  --accept-terms true \
  --profile e2-demo-tokyo

Output:

14:36:13 INFO [d.l.l.transpiler.switch_runner] Uploading /tmp/switch_e2e_test to /Volumes/hinak_catalog_aws_apne1/switch/switch_volume/input-20260130053613-x3za...
14:36:13 INFO [d.l.l.transpiler.switch_runner] Upload complete: /Volumes/hinak_catalog_aws_apne1/switch/switch_volume/input-20260130053613-x3za
14:36:13 INFO [d.l.l.transpiler.switch_runner] Triggering Switch job with job_id: 225510371018306
14:36:13 INFO [d.l.l.transpiler.switch_runner] Switch LLM transpilation job started: https://e2-demo-tokyo.cloud.databricks.com/jobs/225510371018306/runs/556928325013890

Job Parameters (switch_config_path not included):

databricks jobs get-run 556928325013890 --profile e2-demo-tokyo --output json | jq '.job_parameters'

[
  {"name": "source_tech", "value": "mssql"},
  {"name": "input_dir", "value": "/Volumes/hinak_catalog_aws_apne1/switch/switch_volume/input-20260130053613-x3za"},
  {"name": "output_dir", "value": "/Workspace/Users/hiroyuki.nakazato@databricks.com/switch-io/output/test-no-config"},
  {"name": "foundation_model", "value": "databricks-claude-sonnet-4-5"},
  {"name": "catalog", "value": "hinak_catalog_aws_apne1"},
  {"name": "schema", "value": "switch"}
]

Test 2: With --switch-config-path parameter

Command:

databricks labs lakebridge llm-transpile \
  --input-source /tmp/switch_e2e_test \
  --output-ws-folder /Workspace/Users/hiroyuki.nakazato@databricks.com/switch-io/output/test-with-config \
  --source-dialect mssql \
  --catalog-name hinak_catalog_aws_apne1 \
  --schema-name switch \
  --volume switch_volume \
  --foundation-model databricks-claude-sonnet-4-5 \
  --switch-config-path /Workspace/Users/hiroyuki.nakazato@databricks.com/switch_config_test.yml \
  --accept-terms true \
  --profile e2-demo-tokyo

Output:

14:30:39 INFO [d.l.l.transpiler.switch_runner] Uploading /tmp/switch_e2e_test to /Volumes/hinak_catalog_aws_apne1/switch/switch_volume/input-20260130053039-nggi...
14:30:39 INFO [d.l.l.transpiler.switch_runner] Upload complete: /Volumes/hinak_catalog_aws_apne1/switch/switch_volume/input-20260130053039-nggi
14:30:39 INFO [d.l.l.transpiler.switch_runner] Triggering Switch job with job_id: 225510371018306
14:30:40 INFO [d.l.l.transpiler.switch_runner] Switch LLM transpilation job started: https://e2-demo-tokyo.cloud.databricks.com/jobs/225510371018306/runs/242707127988892

Job Parameters (switch_config_path included):

databricks jobs get-run 242707127988892 --profile e2-demo-tokyo --output json | jq '.job_parameters'

[
  {"name": "source_tech", "value": "mssql"},
  {"name": "input_dir", "value": "/Volumes/hinak_catalog_aws_apne1/switch/switch_volume/input-20260130053039-nggi"},
  {"name": "output_dir", "value": "/Workspace/Users/hiroyuki.nakazato@databricks.com/switch-io/output/test-with-config"},
  {"name": "foundation_model", "value": "databricks-claude-sonnet-4-5"},
  {"name": "catalog", "value": "hinak_catalog_aws_apne1"},
  {"name": "schema", "value": "switch"},
  {"name": "switch_config_path", "value": "/Workspace/Users/hiroyuki.nakazato@databricks.com/switch_config_test.yml"}
]

Test 3: Validation error for invalid path

Command:

databricks labs lakebridge llm-transpile \
  --input-source /tmp/switch_e2e_test \
  --output-ws-folder /Workspace/Users/hiroyuki.nakazato@databricks.com/switch-io/output/test \
  --source-dialect mssql \
  --catalog-name hinak_catalog_aws_apne1 \
  --schema-name switch \
  --volume switch_volume \
  --foundation-model databricks-claude-sonnet-4-5 \
  --switch-config-path /Users/invalid/path.yaml \
  --accept-terms true \
  --profile e2-demo-tokyo

Output:

14:31:56 ERROR [d.l.lakebridge.llm-transpile] ValueError: Invalid value for '--switch-config-path': path must start with /Workspace/. Got: '/Users/invalid/path.yaml'

[gueniai]

We need details to the docs as well

[hiroyukinakazato-db]

Documentation updated. @gueniai @asnare @sundarshankar89 Ready for review!

[gueniai]

LGTM

[sundarshankar89]

Before I approve this, can we think about some plan or effort on how to manage these prompts throught mlflow prompt registry. Can we come up with an effort and plan for that activity before exposing these flags to users.