Redshift Profiler PR2: Add Redshift assessment extract SQLs and validation schema

[ysmx-github]

Changes

This PR adds Redshift as a supported profiler assessment platform. It introduces Redshift resources for serverless, provisioned, and provisioned_multi_az

What does this PR do?

• Adds Redshift as a supported platform for the profiler assessment (alongside Synapse).
• Adds Redshift resources for all three variants (serverless, provisioned, provisioned_multi_az): query/SQL files (e.g. query_view, rs_managed_storage_gb, rs_nodes, chart queries), pipeline configs, and validation schema.
• Files (2 levels: folder → subfolder, 3rd level: files under each)):
  • resources/assessments/redshift/ → provisioned/, provisioned_multi_az/, serverless/ with their .sql and pipeline_config.yml
  • resources/assessments/validation/ → redshift_extract_schema.yml

Relevant implementation details

• Resources: Same query/file layout for serverless, provisioned, and provisioned_multi_az under resources/assessments/redshift/{variant}/ (e.g. 0_query_view.sql … 9_chart_*.sql, pipeline_config.yml); serverless also has 10_cost_incurred.sql. Shared validation: resources/assessments/validation/redshift_extract_schema.yml.

Caveats/things to watch out for when reviewing:

• Multi-AZ Redshift has limited system views (e.g. some STV views unavailable); provisioned_multi_az SQL reflects that where applicable.
• Serverless Redshift has one additional query

Linked issues

Resolves #..

Functionality

• added relevant user documentation
• added new CLI command
• modified existing command: databricks labs lakebridge ...
• ... +add your own

Tests

1. Manually Tested all credential flows on all clusters in AWS Sandbox account aws-sandbox-field-eng (332745928618)
   tests/resources/assessments/pipeline_config_main_redshift.yml: pipeline config that runs that script.

• manually tested
• added unit tests
• added integration tests

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.41%. Comparing base (41c3f9a) to head (2d69233).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2304   +/-   ##
=======================================
  Coverage   66.41%   66.41%           
=======================================
  Files          99       99           
  Lines        9094     9094           
  Branches      974      974           
=======================================
  Hits         6040     6040           
  Misses       2878     2878           
  Partials      176      176           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ 145/145 passed, 8 flaky, 4 skipped, 40m34s total

Flaky tests:

• 🤪 test_installs_and_runs_local_bladebridge (22.107s)
• 🤪 test_installs_and_runs_pypi_bladebridge (29.235s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (17.551s)
• 🤪 test_transpiles_informatica_to_sparksql (18.194s)
• 🤪 test_transpile_teradata_sql (19.575s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (4.176s)
• 🤪 test_transpile_teradata_sql_non_interactive[True] (5.787s)
• 🤪 test_transpile_teradata_sql_non_interactive[False] (8.285s)

_{Running from acceptance #4002}

[sundarshankar89]

@ysmx-github i think we should move the chart generation steps to databricks workspace as part of the profiler ingestion job. and show them in our AI/BI dahsboard.

Seperation of duties.

@goodwillpunning Thoughts?

[goodwillpunning]

@ysmx-github i think we should move the chart generation steps to databricks workspace as part of the profiler ingestion job. and show them in our AI/BI dahsboard.

Seperation of duties.

@goodwillpunning Thoughts?

Yes, the charts should be separated out into a dashboard template and visualize-profiler-results will take care of displaying a summary of profiler run to the end user.