Skip to content

feature/oracle_profiler#2187

Open
lolo115 wants to merge 19 commits intomainfrom
feature/oracle_profiler
Open

feature/oracle_profiler#2187
lolo115 wants to merge 19 commits intomainfrom
feature/oracle_profiler

Conversation

@lolo115
Copy link

@lolo115 lolo115 commented Dec 11, 2025

Changes

Creating the oracle extractor.
Unit tests passed for:

  • dialog to configure the extractor
  • run scripts to gather data
  • local database population with extracted data from the Oracle Database.

What does this PR do?

Oracle profiler data extraction

Relevant implementation details

The data analysis is still WIP.

Caveats/things to watch out for when reviewing:

Linked issues

Resolves #..

Functionality

  • added relevant user documentation
  • added new CLI command
  • modified existing command: databricks labs lakebridge ...
  • ... +add your own

Tests

  • manually tested
  • added unit tests
  • added integration tests

@github-actions
Copy link

github-actions bot commented Dec 11, 2025

❌ 139/148 passed, 8 flaky, 9 failed, 4 skipped, 40m31s total

❌ test_Oracle_profile_execution: FileNotFoundError: Configuration file not found for source oracle: Credentials file not found at /home/runner/.databricks/labs/lakebridge/.credentials.yml (321ms)
FileNotFoundError: Configuration file not found for source oracle: Credentials file not found at /home/runner/.databricks/labs/lakebridge/.credentials.yml
[gw7] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
15:36 ERROR [databricks.labs.lakebridge.assessments.profiler] Configuration file not found for source oracle: Credentials file not found at /home/runner/.databricks/labs/lakebridge/.credentials.yml
15:36 ERROR [databricks.labs.lakebridge.assessments.profiler] Configuration file not found for source oracle: Credentials file not found at /home/runner/.databricks/labs/lakebridge/.credentials.yml
[gw7] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
❌ test_profiler_connection_invalid_source_technology: Failed: DID NOT RAISE (1.135s)
Failed: DID NOT RAISE <class 'ValueError'>
[gw9] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra cmd=test-profiler-connection
15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra profiler_source_tech=mssql
15:36 DEBUG [databricks.labs.lakebridge] User: User(active=True, display_name='labs-tool-identity', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')], entitlements=[], external_id='92f40178-1ee9-4156-a6da-28a376a12109', groups=[ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='direct', value='847659649002239'), ComplexValue(display='users', primary=None, ref='Groups/153383108335587', type='direct', value='153383108335587'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='direct', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/149832780896743', type='indirect', value='149832780896743'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='direct', value='688239313962730')], id='1425339244351829', name=Name(family_name=None, given_name='labs-tool-identity'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')
15:36 INFO [databricks.labs.lakebridge] Testing connection for source technology: mssql
15:36 ERROR [databricks.labs.lakebridge] Credential configuration error: 'Source system: mssql credentials not found'
15:36 CRITICAL [databricks.labs.lakebridge] Invalid credentials for mssql. Please run `databricks labs lakebridge configure-database-profiler`.
15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra cmd=test-profiler-connection
15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra profiler_source_tech=mssql
15:36 DEBUG [databricks.labs.lakebridge] User: User(active=True, display_name='labs-tool-identity', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')], entitlements=[], external_id='92f40178-1ee9-4156-a6da-28a376a12109', groups=[ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='direct', value='847659649002239'), ComplexValue(display='users', primary=None, ref='Groups/153383108335587', type='direct', value='153383108335587'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='direct', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/149832780896743', type='indirect', value='149832780896743'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='direct', value='688239313962730')], id='1425339244351829', name=Name(family_name=None, given_name='labs-tool-identity'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')
15:36 INFO [databricks.labs.lakebridge] Testing connection for source technology: mssql
15:36 ERROR [databricks.labs.lakebridge] Credential configuration error: 'Source system: mssql credentials not found'
15:36 CRITICAL [databricks.labs.lakebridge] Invalid credentials for mssql. Please run `databricks labs lakebridge configure-database-profiler`.
[gw9] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
❌ test_profiler_connection_invalid_config_errors: assert 'Missing ODBC driver' in "15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra cmd=test-profiler-connection\n15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra profiler_source_tech=synapse\n15:36 DEBUG [databricks.labs.lakebridge] User: User(active=True, display_name='labs-tool-identity', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')], entitlements=[], external_id='92f40178-1ee9-4156-a6da-28a376a12109', groups=[ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='direct', value='688239313962730'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='direct', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/149832780896743', type='indirect', value='149832780896743'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='direct', value='847659649002239'), ComplexValue(display='users', primary=None, ref='Groups/153383108335587', type='direct', value='153383108335587')], id='1425339244351829', name=Name(family_name=None, given_name='labs-tool-identity'), roles=[], schemas=[, ], user_name='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')\n15:36 INFO [databricks.labs.lakebridge] Testing connection for source technology: synapse\n15:36 INFO [databricks.labs.lakebridge.connections.synapse_connection_helpers] Testing connection to dedicated SQL pool...\n15:36 ERROR [databricks.labs.lakebridge.connections.synapse_connection_helpers] \u2717 Failed to connect to dedicated SQL pool: Unsupported database type: synapse\n15:36 ERROR [databricks.labs.lakebridge] Failed to connect to the source system: Connection failed for SQL pools - dedicated: Failed to connect to dedicated SQL pool: Unsupported database type: synapse\n15:36 CRITICAL [databricks.labs.lakebridge] Connection validation failed. Exiting...\n" (1.017s)
assert 'Missing ODBC driver' in "15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra cmd=test-profiler-connection\n15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra profiler_source_tech=synapse\n15:36 DEBUG [databricks.labs.lakebridge] User: User(active=True, display_name='labs-tool-identity', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')], entitlements=[], external_id='92f40178-1ee9-4156-a6da-28a376a12109', groups=[ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='direct', value='688239313962730'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='direct', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/149832780896743', type='indirect', value='149832780896743'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='direct', value='847659649002239'), ComplexValue(display='users', primary=None, ref='Groups/153383108335587', type='direct', value='153383108335587')], id='1425339244351829', name=Name(family_name=None, given_name='labs-tool-identity'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')\n15:36 INFO [databricks.labs.lakebridge] Testing connection for source technology: synapse\n15:36 INFO [databricks.labs.lakebridge.connections.synapse_connection_helpers] Testing connection to dedicated SQL pool...\n15:36 ERROR [databricks.labs.lakebridge.connections.synapse_connection_helpers] \u2717 Failed to connect to dedicated SQL pool: Unsupported database type: synapse\n15:36 ERROR [databricks.labs.lakebridge] Failed to connect to the source system: Connection failed for SQL pools - dedicated: Failed to connect to dedicated SQL pool: Unsupported database type: synapse\n15:36 CRITICAL [databricks.labs.lakebridge] Connection validation failed. Exiting...\n"
 +  where "15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra cmd=test-profiler-connection\n15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra profiler_source_tech=synapse\n15:36 DEBUG [databricks.labs.lakebridge] User: User(active=True, display_name='labs-tool-identity', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')], entitlements=[], external_id='92f40178-1ee9-4156-a6da-28a376a12109', groups=[ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='direct', value='688239313962730'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='direct', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/149832780896743', type='indirect', value='149832780896743'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='direct', value='847659649002239'), ComplexValue(display='users', primary=None, ref='Groups/153383108335587', type='direct', value='153383108335587')], id='1425339244351829', name=Name(family_name=None, given_name='labs-tool-identity'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')\n15:36 INFO [databricks.labs.lakebridge] Testing connection for source technology: synapse\n15:36 INFO [databricks.labs.lakebridge.connections.synapse_connection_helpers] Testing connection to dedicated SQL pool...\n15:36 ERROR [databricks.labs.lakebridge.connections.synapse_connection_helpers] \u2717 Failed to connect to dedicated SQL pool: Unsupported database type: synapse\n15:36 ERROR [databricks.labs.lakebridge] Failed to connect to the source system: Connection failed for SQL pools - dedicated: Failed to connect to dedicated SQL pool: Unsupported database type: synapse\n15:36 CRITICAL [databricks.labs.lakebridge] Connection validation failed. Exiting...\n" = <_pytest.logging.LogCaptureFixture object at 0x7f4b04c59990>.text
[gw9] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra cmd=test-profiler-connection
15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra profiler_source_tech=synapse
15:36 DEBUG [databricks.labs.lakebridge] User: User(active=True, display_name='labs-tool-identity', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')], entitlements=[], external_id='92f40178-1ee9-4156-a6da-28a376a12109', groups=[ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='direct', value='688239313962730'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='direct', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/149832780896743', type='indirect', value='149832780896743'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='direct', value='847659649002239'), ComplexValue(display='users', primary=None, ref='Groups/153383108335587', type='direct', value='153383108335587')], id='1425339244351829', name=Name(family_name=None, given_name='labs-tool-identity'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')
15:36 INFO [databricks.labs.lakebridge] Testing connection for source technology: synapse
15:36 INFO [databricks.labs.lakebridge.connections.synapse_connection_helpers] Testing connection to dedicated SQL pool...
15:36 ERROR [databricks.labs.lakebridge.connections.synapse_connection_helpers] ✗ Failed to connect to dedicated SQL pool: Unsupported database type: synapse
15:36 ERROR [databricks.labs.lakebridge] Failed to connect to the source system: Connection failed for SQL pools - dedicated: Failed to connect to dedicated SQL pool: Unsupported database type: synapse
15:36 CRITICAL [databricks.labs.lakebridge] Connection validation failed. Exiting...
15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra cmd=test-profiler-connection
15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra profiler_source_tech=synapse
15:36 DEBUG [databricks.labs.lakebridge] User: User(active=True, display_name='labs-tool-identity', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')], entitlements=[], external_id='92f40178-1ee9-4156-a6da-28a376a12109', groups=[ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='direct', value='688239313962730'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='direct', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/149832780896743', type='indirect', value='149832780896743'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='direct', value='847659649002239'), ComplexValue(display='users', primary=None, ref='Groups/153383108335587', type='direct', value='153383108335587')], id='1425339244351829', name=Name(family_name=None, given_name='labs-tool-identity'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')
15:36 INFO [databricks.labs.lakebridge] Testing connection for source technology: synapse
15:36 INFO [databricks.labs.lakebridge.connections.synapse_connection_helpers] Testing connection to dedicated SQL pool...
15:36 ERROR [databricks.labs.lakebridge.connections.synapse_connection_helpers] ✗ Failed to connect to dedicated SQL pool: Unsupported database type: synapse
15:36 ERROR [databricks.labs.lakebridge] Failed to connect to the source system: Connection failed for SQL pools - dedicated: Failed to connect to dedicated SQL pool: Unsupported database type: synapse
15:36 CRITICAL [databricks.labs.lakebridge] Connection validation failed. Exiting...
[gw9] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
❌ test_synapse_connector_execute_query: ValueError: Unsupported database type: synapse (8ms)
ValueError: Unsupported database type: synapse
[gw4] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
❌ test_synapse_connection_check: ValueError: Unsupported database type: synapse (4ms)
ValueError: Unsupported database type: synapse
[gw4] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
❌ test_synapse_with_credential_format: ValueError: Unsupported database type: synapse (5ms)
ValueError: Unsupported database type: synapse
[gw4] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
[gw4] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
❌ test_synapse_query_execution: ValueError: Unsupported database type: synapse (7ms)
ValueError: Unsupported database type: synapse
[gw4] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
[gw4] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
❌ test_profiler_connection_synapse_success: assert '\u2713 Dedicated SQL pool connection successful' in "15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra cmd=test-profiler-connection\n15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra profiler_source_tech=synapse\n15:36 DEBUG [databricks.labs.lakebridge] User: User(active=True, display_name='labs-tool-identity', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')], entitlements=[], external_id='92f40178-1ee9-4156-a6da-28a376a12109', groups=[ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='direct', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/149832780896743', type='indirect', value='149832780896743'), ComplexValue(display='users', primary=None, ref='Groups/153383108335587', type='direct', value='153383108335587'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='direct', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='direct', value='688239313962730')], id='1425339244351829', name=Name(family_name=None, given_name='labs-tool-identity'), roles=[], schemas=[, ], user_name='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')\n15:36 INFO [databricks.labs.lakebridge] Testing connection for source technology: synapse\n15:36 INFO [databricks.labs.lakebridge.connections.synapse_connection_helpers] Testing connection to dedicated SQL pool...\n15:36 ERROR [databricks.labs.lakebridge.connections.synapse_connection_helpers] \u2717 Failed to connect to dedicated SQL pool: Unsupported database type: synapse\n15:36 ERROR [databricks.labs.lakebridge] Failed to connect to the source system: Connection failed for SQL pools - dedicated: Failed to connect to dedicated SQL pool: Unsupported database type: synapse\n15:36 CRITICAL [databricks.labs.lakebridge] Connection validation failed. Exiting...\n" (1.119s)
assert '\u2713 Dedicated SQL pool connection successful' in "15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra cmd=test-profiler-connection\n15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra profiler_source_tech=synapse\n15:36 DEBUG [databricks.labs.lakebridge] User: User(active=True, display_name='labs-tool-identity', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')], entitlements=[], external_id='92f40178-1ee9-4156-a6da-28a376a12109', groups=[ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='direct', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/149832780896743', type='indirect', value='149832780896743'), ComplexValue(display='users', primary=None, ref='Groups/153383108335587', type='direct', value='153383108335587'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='direct', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='direct', value='688239313962730')], id='1425339244351829', name=Name(family_name=None, given_name='labs-tool-identity'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')\n15:36 INFO [databricks.labs.lakebridge] Testing connection for source technology: synapse\n15:36 INFO [databricks.labs.lakebridge.connections.synapse_connection_helpers] Testing connection to dedicated SQL pool...\n15:36 ERROR [databricks.labs.lakebridge.connections.synapse_connection_helpers] \u2717 Failed to connect to dedicated SQL pool: Unsupported database type: synapse\n15:36 ERROR [databricks.labs.lakebridge] Failed to connect to the source system: Connection failed for SQL pools - dedicated: Failed to connect to dedicated SQL pool: Unsupported database type: synapse\n15:36 CRITICAL [databricks.labs.lakebridge] Connection validation failed. Exiting...\n"
 +  where "15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra cmd=test-profiler-connection\n15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra profiler_source_tech=synapse\n15:36 DEBUG [databricks.labs.lakebridge] User: User(active=True, display_name='labs-tool-identity', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')], entitlements=[], external_id='92f40178-1ee9-4156-a6da-28a376a12109', groups=[ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='direct', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/149832780896743', type='indirect', value='149832780896743'), ComplexValue(display='users', primary=None, ref='Groups/153383108335587', type='direct', value='153383108335587'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='direct', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='direct', value='688239313962730')], id='1425339244351829', name=Name(family_name=None, given_name='labs-tool-identity'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')\n15:36 INFO [databricks.labs.lakebridge] Testing connection for source technology: synapse\n15:36 INFO [databricks.labs.lakebridge.connections.synapse_connection_helpers] Testing connection to dedicated SQL pool...\n15:36 ERROR [databricks.labs.lakebridge.connections.synapse_connection_helpers] \u2717 Failed to connect to dedicated SQL pool: Unsupported database type: synapse\n15:36 ERROR [databricks.labs.lakebridge] Failed to connect to the source system: Connection failed for SQL pools - dedicated: Failed to connect to dedicated SQL pool: Unsupported database type: synapse\n15:36 CRITICAL [databricks.labs.lakebridge] Connection validation failed. Exiting...\n" = <_pytest.logging.LogCaptureFixture object at 0x7f36fcb65d50>.text
[gw6] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra cmd=test-profiler-connection
15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra profiler_source_tech=synapse
15:36 DEBUG [databricks.labs.lakebridge] User: User(active=True, display_name='labs-tool-identity', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')], entitlements=[], external_id='92f40178-1ee9-4156-a6da-28a376a12109', groups=[ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='direct', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/149832780896743', type='indirect', value='149832780896743'), ComplexValue(display='users', primary=None, ref='Groups/153383108335587', type='direct', value='153383108335587'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='direct', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='direct', value='688239313962730')], id='1425339244351829', name=Name(family_name=None, given_name='labs-tool-identity'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')
15:36 INFO [databricks.labs.lakebridge] Testing connection for source technology: synapse
15:36 INFO [databricks.labs.lakebridge.connections.synapse_connection_helpers] Testing connection to dedicated SQL pool...
15:36 ERROR [databricks.labs.lakebridge.connections.synapse_connection_helpers] ✗ Failed to connect to dedicated SQL pool: Unsupported database type: synapse
15:36 ERROR [databricks.labs.lakebridge] Failed to connect to the source system: Connection failed for SQL pools - dedicated: Failed to connect to dedicated SQL pool: Unsupported database type: synapse
15:36 CRITICAL [databricks.labs.lakebridge] Connection validation failed. Exiting...
15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra cmd=test-profiler-connection
15:36 DEBUG [databricks.labs.lakebridge.contexts.application] Added User-Agent extra profiler_source_tech=synapse
15:36 DEBUG [databricks.labs.lakebridge] User: User(active=True, display_name='labs-tool-identity', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')], entitlements=[], external_id='92f40178-1ee9-4156-a6da-28a376a12109', groups=[ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='direct', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/149832780896743', type='indirect', value='149832780896743'), ComplexValue(display='users', primary=None, ref='Groups/153383108335587', type='direct', value='153383108335587'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='direct', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='direct', value='688239313962730')], id='1425339244351829', name=Name(family_name=None, given_name='labs-tool-identity'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='3fe685a1-96cc-4fec-8cdb-6944f5c9787e')
15:36 INFO [databricks.labs.lakebridge] Testing connection for source technology: synapse
15:36 INFO [databricks.labs.lakebridge.connections.synapse_connection_helpers] Testing connection to dedicated SQL pool...
15:36 ERROR [databricks.labs.lakebridge.connections.synapse_connection_helpers] ✗ Failed to connect to dedicated SQL pool: Unsupported database type: synapse
15:36 ERROR [databricks.labs.lakebridge] Failed to connect to the source system: Connection failed for SQL pools - dedicated: Failed to connect to dedicated SQL pool: Unsupported database type: synapse
15:36 CRITICAL [databricks.labs.lakebridge] Connection validation failed. Exiting...
[gw6] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python
❌ test_synapse_connector_connection: ValueError: Unsupported database type: synapse (11ms)
ValueError: Unsupported database type: synapse
[gw7] linux -- Python 3.10.19 /home/runner/work/lakebridge/lakebridge/.venv/bin/python

Flaky tests:

  • 🤪 test_installs_and_runs_local_bladebridge (20.886s)
  • 🤪 test_installs_and_runs_pypi_bladebridge (25.761s)
  • 🤪 test_transpiles_informatica_to_sparksql (19.45s)
  • 🤪 test_transpile_teradata_sql_non_interactive[True] (21.478s)
  • 🤪 test_transpile_teradata_sql (21.657s)
  • 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (4.194s)
  • 🤪 test_transpile_teradata_sql_non_interactive[False] (5.983s)
  • 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (13.859s)

Running from acceptance #4008

@codecov
Copy link

codecov bot commented Dec 11, 2025

Codecov Report

❌ Patch coverage is 28.57143% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.32%. Comparing base (41c3f9a) to head (f99b785).

Files with missing lines Patch % Lines
...abs/lakebridge/assessments/configure_assessment.py 16.66% 10 Missing ⚠️
...ks/labs/lakebridge/connections/database_manager.py 37.50% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2187      +/-   ##
==========================================
- Coverage   66.41%   66.32%   -0.10%     
==========================================
  Files          99       99              
  Lines        9094     9114      +20     
  Branches      974      975       +1     
==========================================
+ Hits         6040     6045       +5     
- Misses       2878     2893      +15     
  Partials      176      176              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Updates on sql scripts to remove spooling "decoration"
# Conflicts:
#	pyproject.toml
#	src/databricks/labs/lakebridge/connections/database_manager.py
# Conflicts:
#	pyproject.toml
#	src/databricks/labs/lakebridge/connections/database_manager.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants