Improve Create Profiler Dashboard CLI Usage by goodwillpunning · Pull Request #2319 · databrickslabs/lakebridge

goodwillpunning · 2026-03-01T23:58:25Z

Changes

What does this PR do?

This PR updates the deployment of the profiler summary dashboard to be more consistent with other Lakebridge components, such as the recon job and dashboards.

Relevant implementation details

Improved CLI prompts so that the extract file location, UC catalog, schema, and volume name are more clear
Adds a helper function to properly parse the extract file location and UC volume upload location
Hooks the installation of the profiler dashboard into the Lakebridge installer/uninstaller components

Caveats/things to watch out for when reviewing:

Linked issues

N/A

Functionality

added relevant user documentation
added new CLI command
modified existing command: databricks labs lakebridge ...
... +add your own

Tests

manually tested
added unit tests
added integration tests

github-actions · 2026-03-02T00:19:28Z

✅ 145/145 passed, 7 flaky, 4 skipped, 37m39s total

Flaky tests:

🤪 test_installs_and_runs_local_bladebridge (19.248s)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (14.378s)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (15.033s)
🤪 test_transpiles_informatica_to_sparksql (15.097s)
🤪 test_transpile_teradata_sql_non_interactive[True] (4.487s)
🤪 test_transpile_teradata_sql_non_interactive[False] (4.436s)
🤪 test_transpile_teradata_sql (5.014s)

_{Running from acceptance #4011}

sundarshankar89

Let us take a step back, I think the approach needs to be composable

I would approach this as two commands

databricks labs lakebridge profiler-sync

databricks labs lakebridge deploy-profiler-dashboard

profiler-sync is pre-req for deploy-profiler-dashboard

profiler-sync Creates ingestion job and syncs the data into profiler tables within databricks. it is intentionally called sync because it creates infra if not already exists and triggers incremental data load when applied.

deploy-profiler-dashboard just creates AI/BI dashboard one of so that it points agains the tables or objects fails if the object doesn't exists and request user to run profiler-sync before running deploy-profiler-dashboard

Thoughts?

src/databricks/labs/lakebridge/deployment/installation.py

sundarshankar89 · 2026-03-02T07:37:49Z

src/databricks/labs/lakebridge/deployment/dashboard.py

+        logging.info(f"Loading dashboard template from folder: {folder}")
+        dash_reference = f"{folder.stem}".lower()
+        dashboard_loader = ProfilerDashboardTemplateLoader(folder)
+        dashboard_json = dashboard_loader.load(source_system="synapse")


source system needs to be argument.

Great catch!

sundarshankar89 · 2026-03-02T07:40:14Z

src/databricks/labs/lakebridge/deployment/profiler_dashboard.py

+            logger.warning("Profiler Dashboard Config is empty.")
+            return
+        logger.info("Installing the profiler dashboard components.")
+        self._upload_profiler_extract(profiler_dashboard_config)


need some error handling here, for catching upload errors and permission checks.

Agreed. Added better error handling to catch those exceptions.

src/databricks/labs/lakebridge/cli.py

sundarshankar89 · 2026-03-02T08:09:25Z

src/databricks/labs/lakebridge/deployment/profiler_dashboard.py

+        logger.info("Uninstalling profiler dashboard components.")
+        self._remove_dashboards()
+        self._remove_jobs()
+        logging.info(


Suggested change

logging.info(

logger.info(

sundarshankar89 · 2026-03-02T08:10:29Z

src/databricks/labs/lakebridge/deployment/profiler_dashboard.py

+        self._installation = installation
+        self._install_state = install_state
+        self._product_info = product_info
+        self._table_deployer = table_deployer


I don't see it used anywhere may be we can remove it

Suggested change

self._table_deployer = table_deployer

Good catch. Removed.

src/databricks/labs/lakebridge/deployment/dashboard.py

codecov · 2026-03-03T18:37:07Z

Codecov Report

❌ Patch coverage is 57.80731% with 127 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.68%. Comparing base (41c3f9a) to head (9ed2790).

Files with missing lines	Patch %	Lines
...s/labs/lakebridge/deployment/profiler_dashboard.py	34.88%	56 Missing ⚠️
...databricks/labs/lakebridge/deployment/dashboard.py	50.00%	50 Missing and 3 partials ⚠️
src/databricks/labs/lakebridge/install.py	75.67%	8 Missing and 1 partial ⚠️
src/databricks/labs/lakebridge/cli.py	11.11%	8 Missing ⚠️
.../labs/lakebridge/assessments/dashboards/execute.py	91.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2319      +/-   ##
==========================================
+ Coverage   66.41%   67.68%   +1.26%     
==========================================
  Files          99       99              
  Lines        9094     9271     +177     
  Branches      974      986      +12     
==========================================
+ Hits         6040     6275     +235     
+ Misses       2878     2816      -62     
- Partials      176      180       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sundarshankar89

@goodwillpunning everything looks good, there is small improvement I would like to do, which is make the job deployer and dashboard deployer generic so it can be used by both profiler and reconcile job. We can iterate on this do not want block this PR.

sundarshankar89 · 2026-03-06T06:19:53Z

docs/lakebridge/docs/assessment/profiler/dashboards/index.mdx

+| Amazon Redshift | &#x274C; |
+| Oracle | &#x274C; |
+| Microsoft SQL Server | &#x274C; |
+| Snowflake | &#x274C; |


should we leave it here, may be better to update later or changing to coming soon.

sundarshankar89 · 2026-03-06T06:20:56Z

src/databricks/labs/lakebridge/assessments/dashboards/execute.py



-def _ingest_table(extract_location: str, source_table_name: str, target_table_name: str) -> None:
+def ingest_table(extract_location: str, source_table_name: str, target_table_name: str) -> None:


@goodwillpunning is there reason we change all the method signature from private to public?

To make it easy to unit test without disabling pylint ( # pylint: disable=import-private-name forbidden by the build check).

sundarshankar89 · 2026-03-06T11:38:12Z

src/databricks/labs/lakebridge/deployment/dashboard.py

+        if not os.path.exists(filepath):
+            raise FileNotFoundError(f"Could not find dashboard template matching {source_system}.")
+        with open(filepath, "r", encoding="utf-8") as f:
+            return json.load(f)


Suggested change

if not os.path.exists(filepath):

raise FileNotFoundError(f"Could not find dashboard template matching {source_system}.")

with open(filepath, "r", encoding="utf-8") as f:

return json.load(f)

try:

with open(filepath, "r", encoding="utf-8") as f:

return json.load(f)

except FileNotFoundError:

raise FileNotFoundError(f"Could not find dashboard template matching {source_system}.")

sundarshankar89 · 2026-03-06T11:38:28Z

src/databricks/labs/lakebridge/deployment/dashboard.py

+        """
+
+        # Load the dashboard template
+        logging.info(f"Loading dashboard template from folder: {folder}")


Suggested change

logging.info(f"Loading dashboard template from folder: {folder}")

logger.info(f"Loading dashboard template from folder: {folder}")

sundarshankar89 · 2026-03-06T11:38:39Z

src/databricks/labs/lakebridge/deployment/dashboard.py

+        try:
+            dashboard = self._ws.lakeview.create(dashboard=dashboard)
+        except ResourceAlreadyExists:
+            logging.info("Dashboard already exists! Removing dashboard from workspace location.")


Suggested change

logging.info("Dashboard already exists! Removing dashboard from workspace location.")

logger.info("Dashboard already exists! Removing dashboard from workspace location.")

src/databricks/labs/lakebridge/install.py

sundarshankar89 · 2026-03-06T11:49:23Z

src/databricks/labs/lakebridge/install.py

+    def _configure_new_profiler_dashboard_installation(self) -> ProfilerDashboardConfig:
+        default_config = self._prompt_for_new_profiler_dashboard_installation()
+        self._save_config(default_config)
+        return default_config
+


no need of this

Suggested change

def _configure_new_profiler_dashboard_installation(self) -> ProfilerDashboardConfig:

default_config = self._prompt_for_new_profiler_dashboard_installation()

self._save_config(default_config)

return default_config

sundarshankar89 · 2026-03-06T11:50:24Z

src/databricks/labs/lakebridge/deployment/profiler_dashboard.py

+        return [
+            (job_name, int(job_id))
+            for job_name, job_id in self._install_state.jobs.items()
+            if job_name.startswith(_PROFILER_DASHBOARD_PREFIX) and job_name != PROFILER_INGESTION_JOB_NAME
+        ]


Suggested change

return [

(job_name, int(job_id))

for job_name, job_id in self._install_state.jobs.items()

if job_name.startswith(_PROFILER_DASHBOARD_PREFIX) and job_name != PROFILER_INGESTION_JOB_NAME

]

return [(name, job_id) for name, job_id in self._get_jobs() if name != PROFILER_INGESTION_JOB_NAME]

sundarshankar89 · 2026-03-06T11:51:42Z

tests/unit/deployment/test_dashboard_manager.py

 def test_upload_duckdb_to_uc_volume_invalid_volume_path(
-    dashboard_manager: DashboardManager,
+    dashboard_manager: ProfilerDashboardManager,
    mocked_workspace_client: WorkspaceClient,
 ):
    ws = mocked_workspace_client
-    result = dashboard_manager.upload_duckdb_to_uc_volume(
-        local_file_path="file.duckdb", volume_path="invalid_path/myfile.duckdb"
+    config = ProfilerDashboardConfig(
+        source_tech="synapse",
+        extract_file_path="file.duckdb",
+        metadata_config=ProfilerDashboardMetadataConfig(catalog="lakebridge", schema="profiler", volume="invalid_path"),
    )
+    result = dashboard_manager.upload_duckdb_to_uc_volume(config)
    assert result is False
    ws.files.upload.assert_not_called()



we no longer need this test since we dont branch and test this particular path in dashboard_manager.py

Co-authored-by: SundarShankar89 <72757199+sundarshankar89@users.noreply.github.com>

gueniai

LGTM

sundarshankar89 · 2026-03-11T05:44:14Z

@goodwillpunning can you resolve conflicts then this can be ready to merge

goodwillpunning added 2 commits March 1, 2026 18:02

Update create-profiler-dashboard CLI args.

45ebe67

Update profiler dashboard config and job params.

c1b8b0c

goodwillpunning self-assigned this Mar 1, 2026

goodwillpunning requested a review from a team as a code owner March 1, 2026 23:58

goodwillpunning added the feat/profiler Issues related to profilers label Mar 1, 2026

goodwillpunning had a problem deploying to tool March 1, 2026 23:58 — with GitHub Actions Failure

goodwillpunning requested review from gueniai and sundarshankar89 March 1, 2026 23:58

Move profiler dashboard deployment into installer.

4a371d3

goodwillpunning had a problem deploying to tool March 2, 2026 00:36 — with GitHub Actions Error

Fix failing tests.

8765bb5

goodwillpunning had a problem deploying to tool March 2, 2026 00:47 — with GitHub Actions Error

Fix test conf and formatting.

0e788ed

goodwillpunning had a problem deploying to tool March 2, 2026 00:52 — with GitHub Actions Failure

Update extract validation check.

214a951

goodwillpunning had a problem deploying to tool March 2, 2026 04:43 — with GitHub Actions Failure

Fix closed connection error.

30e0afe

goodwillpunning had a problem deploying to tool March 2, 2026 05:16 — with GitHub Actions Failure

sundarshankar89 requested changes Mar 2, 2026

View reviewed changes

sundarshankar89 mentioned this pull request Mar 2, 2026

Update Profiler Dashboard Templates #2317

Merged

7 tasks

Add doc page for publishing profiler summary dashboard.

3d49b76

goodwillpunning temporarily deployed to tool March 3, 2026 16:50 — with GitHub Actions Inactive

Rename create-profiler-dashboard to visualize-profiler-results

745e0f3

goodwillpunning temporarily deployed to tool March 3, 2026 17:19 — with GitHub Actions Inactive

Fix failing unit test.

3d95680

goodwillpunning temporarily deployed to tool March 3, 2026 18:11 — with GitHub Actions Inactive

Add unit tests for profiler ingestion job.

d775dcd

goodwillpunning had a problem deploying to tool March 4, 2026 15:45 — with GitHub Actions Error

Add missing param for source system.

3618045

goodwillpunning had a problem deploying to tool March 4, 2026 15:51 — with GitHub Actions Error

Update visualize-profiler-results command description.

9dbae8d

goodwillpunning had a problem deploying to tool March 4, 2026 15:56 — with GitHub Actions Error

Remove unused table_deployment argument.

6bd7c75

goodwillpunning had a problem deploying to tool March 4, 2026 16:01 — with GitHub Actions Error

Hook profiler uninstallation functions into global uninstaller.

9ade66e

goodwillpunning had a problem deploying to tool March 4, 2026 16:04 — with GitHub Actions Error

Update helper functions to be publicly accessible.

cfc6609

goodwillpunning temporarily deployed to tool March 4, 2026 16:17 — with GitHub Actions Inactive

Improve installation error handling and fix failing unit test.

0b492aa

goodwillpunning temporarily deployed to tool March 4, 2026 17:09 — with GitHub Actions Inactive

Update dashboard template file path construction.

50df211

goodwillpunning temporarily deployed to tool March 4, 2026 18:51 — with GitHub Actions Inactive

Fix formatting.

9c8f10b

goodwillpunning temporarily deployed to tool March 4, 2026 19:53 — with GitHub Actions Inactive

Merge branch 'main' into fix/profiler_cli_usage

a145f74

goodwillpunning had a problem deploying to tool March 5, 2026 16:12 — with GitHub Actions Failure

sundarshankar89 reviewed Mar 6, 2026

View reviewed changes

Update Oracle, MSSQL, and Snowflake sources as coming soon

2a6d1ff

goodwillpunning temporarily deployed to tool March 6, 2026 16:13 — with GitHub Actions Inactive

goodwillpunning added 2 commits March 6, 2026 11:15

Update logging statements to use global logger.

6c44894

Refactor file not found guard.

8b6e56c

goodwillpunning had a problem deploying to tool March 6, 2026 16:43 — with GitHub Actions Error

Apply suggestion from @sundarshankar89

9ed2790

Co-authored-by: SundarShankar89 <72757199+sundarshankar89@users.noreply.github.com>

goodwillpunning temporarily deployed to tool March 6, 2026 16:47 — with GitHub Actions Inactive

goodwillpunning and others added 2 commits March 6, 2026 14:25

Apply suggestion from @sundarshankar89

dfaf360

Co-authored-by: SundarShankar89 <72757199+sundarshankar89@users.noreply.github.com>

Fix missplaced try-except opening.

ad259ff

gueniai approved these changes Mar 6, 2026

View reviewed changes



		def _ingest_table(extract_location: str, source_table_name: str, target_table_name: str) -> None:
		def ingest_table(extract_location: str, source_table_name: str, target_table_name: str) -> None:

	logging.info(f"Loading dashboard template from folder: {folder}")
	logger.info(f"Loading dashboard template from folder: {folder}")

	logging.info("Dashboard already exists! Removing dashboard from workspace location.")
	logger.info("Dashboard already exists! Removing dashboard from workspace location.")

Conversation

goodwillpunning commented Mar 1, 2026

Changes

What does this PR do?

Relevant implementation details

Caveats/things to watch out for when reviewing:

Linked issues

Functionality

Tests

Uh oh!

github-actions bot commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sundarshankar89 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sundarshankar89 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gueniai left a comment

Choose a reason for hiding this comment

Uh oh!

sundarshankar89 commented Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Mar 2, 2026 •

edited

Loading

codecov bot commented Mar 3, 2026 •

edited

Loading