PLUGIN-1823: Retrying all SQLTransientExceptions #597

sgarg-CS · 2025-05-16T11:36:26Z

Add Failsafe Retry poilcy to all the places in the database-plugins where SQLTransientException could be thrown.

Added three new properties (hidden from UI)

Initial Retry Duration (Default: 5sec)
Max Retry Duration (Default: 80 sec)
Max Retry Count (Default: 5)

google-cla · 2025-05-16T11:36:31Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

itsankit-google

Overall, this looks everything is getting wrapped within Failsafe where we might end up with having nested level retries, we need to ensure we add retries only where we are actually interacting with JDBC client and not top level functions.

For example adding retries to DriverManager.getConnection(connectionString, connectionProperties) makes sense because you are actually interacting with the source db but adding retries to whole loadSchema(Connection connection, String query) do not makes sense we need to be careful while adding such retries.

database-commons/src/main/java/io/cdap/plugin/db/action/DBRun.java

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

itsankit-google · 2025-05-30T05:31:01Z

Please note E2E should not be modified and not fail with these changes. Otherwise, we have done something wrong which does not give expected failure messages.

database-commons/src/main/java/io/cdap/plugin/db/sink/AbstractDBSink.java

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

itsankit-google

I can see some level of duplication in both AbstractDBSource & AbstractDBSink, can we please move it to the common AbstractDBUtil class?

database-commons/src/main/java/io/cdap/plugin/db/ConnectionConfig.java

cloudsql-mysql-plugin/src/test/java/io/cdap/plugin/cloudsql/mysql/CloudSQLMySQLSinkTest.java

database-commons/src/main/java/io/cdap/plugin/db/connector/AbstractDBSpecificConnector.java

itsankit-google

public final class RetryUtils {

  public static Connection createConnectionWithRetry(RetryPolicy<?> retryPolicy, String connectionString,
                                                     Properties connectionProperties, String externalDocumentationLink) throws Exception {
    try {
      return Failsafe.with(retryPolicy).get(() ->
        DriverManager.getConnection(connectionString, connectionProperties)
      );
    } catch (Exception e) {
      throw unwrapFailsafeException(e, externalDocumentationLink);
    }
  }

  public static Statement createStatementWithRetry(RetryPolicy<?> retryPolicy,
                                                   Connection connection, String externalDocumentationLink) throws Exception {
    try {
      return Failsafe.with(retryPolicy).get(connection::createStatement);
    } catch (Exception e) {
      throw unwrapFailsafeException(e, externalDocumentationLink);
    }
  }

  public static PreparedStatement prepareStatementWithRetry(RetryPolicy<?> retryPolicy,
                                                            Connection connection,
                                                            String sqlQuery, String externalDocumentationLink) throws Exception {
    try {
      return Failsafe.with(retryPolicy).get(() ->
        connection.prepareStatement(sqlQuery)
      );
    } catch (Exception e) {
      throw unwrapFailsafeException(e, externalDocumentationLink);
    }
  }

 public static ResultSet executeWithRetry(RetryPolicy<?> retryPolicy,
                                                            Connection connection,
                                                            String sqlQuery, String externalDocumentationLink) throws Exception {
        try {
            return Failsafe.with(retryPolicy).get(() -> connection.createStatement().executeQuery(sqlQuery));
        } catch (Exception e) {
            throw unwrapFailsafeException(e, externalDocumentationLink);
        }
    }

 private static Exception unwrapFailsafeException(Exception e) {
    if (e instanceof FailsafeException && e.getCause() instanceof Exception) {
        if (e instanceOf SQLException) {
           return programFailureException(e, externalDocumentationLink);
        } else {
          return (Exception) e.getCause();
       }
    }
    return e;
  }
  
private static ProgramFailureException programFailureException(SQLException e, String externalDocumentationLink) {
    // wrap exception to ensure SQLException-child instances not exposed to contexts without jdbc
    // driver in classpath
    String errorMessage =
      String.format("SQL Exception occurred: [Message='%s', SQLState='%s', ErrorCode='%s'].",
        e.getMessage(), e.getSQLState(), e.getErrorCode());
    String errorMessageWithDetails = String.format("Error occurred while trying to" +
      " get schema from database." + "Error message: '%s'. Error code: '%s'. SQLState: '%s'", e.getMessage(),
        e.getErrorCode(), e.getSQLState());
 
    if (!Strings.isNullOrEmpty(externalDocumentationLink)) {
      if (!errorMessage.endsWith(".")) {
        errorMessage = errorMessage + ".";
      }
      errorMessage = String.format("%s For more details, see %s", errorMessage, externalDocumentationLink);
    }
    return ErrorUtils.getProgramFailureException(new ErrorCategory(ErrorCategory.ErrorCategoryEnum.PLUGIN),
      errorMessage, errorMessageWithDetails, ErrorType.USER, false, ErrorCodeType.SQLSTATE, e.getSQLState(),
        externalDocumentationLink, e);
  }
}

You can create a RetryUtils like above which accepts connection params.

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

sgarg-CS · 2025-06-03T14:04:42Z

Overall, this looks everything is getting wrapped within Failsafe where we might end up with having nested level retries, we need to ensure we add retries only where we are actually interacting with JDBC client and not top level functions.

For example adding retries to DriverManager.getConnection(connectionString, connectionProperties) makes sense because you are actually interacting with the source db but adding retries to whole loadSchema(Connection connection, String query) do not makes sense we need to be careful while adding such retries.

Refactored the code to add the retry logic only for the methods interacting with the JDBC client.

database-commons/src/main/java/io/cdap/plugin/util/RetryUtils.java

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

database-commons/src/main/java/io/cdap/plugin/util/RetryUtils.java

database-commons/src/main/java/io/cdap/plugin/db/connector/AbstractDBConnectorConfig.java

database-commons/src/main/java/io/cdap/plugin/db/sink/AbstractDBSink.java

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

itsankit-google

How does it affect the UI if validateSchema fails for DB sinks? Can we test before/after the change?

sgarg-CS · 2025-06-05T07:23:50Z

How does it affect the UI if validateSchema fails for DB sinks? Can we test before/after the change?

Yes. Will check and update the behaviour here.

I've reverted the changes done to handle SQLException thrown by validateSchema() method call from AbstractDBSink.configurePipeline class. Still, I see a change in the error message on the UI. This is probably due to catching SQLException and then wrapping it to throw ProgramFailureException. While this wrapping was done for inferSchema() but not for validateSchema() in AbstractSink class earlier.

Test Scenario: Validate the schema in PostgreSQL Sink Plugin, if connection is not active. (Postgres DB is down)

[BEFORE CHANGES]

Error message on the UI:
Exception while trying to validate schema of database table '"users2"' for connection 'jdbc:postgresql://localhost:5433/postgres' with Connection to localhost:5433 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

[AFTER CHANGES]

Error message on the UI:
Error encountered while configuring the stage: 'Error occurred while trying to get schema from database. Error message: 'Connection to localhost:5433 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.'. Error code: '0'. SQLState: '08001''

Is the new error message acceptable or should we revert the changes done to throw ProgramFailureException for validateSchema() method?

itsankit-google · 2025-06-05T13:07:05Z

How does it affect the UI if validateSchema fails for DB sinks? Can we test before/after the change?

Yes. Will check and update the behaviour here.

I've reverted the changes done to handle SQLException thrown by validateSchema() method call from AbstractDBSink.configurePipeline class. Still, I see a change in the error message on the UI. This is probably due to catching SQLException and then wrapping it to throw ProgramFailureException. While this wrapping was done for inferSchema() but not for validateSchema() in AbstractSink class earlier.

Test Scenario: Validate the schema in PostgreSQL Sink Plugin, if connection is not active. (Postgres DB is down)

[BEFORE CHANGES]

Error message on the UI: Exception while trying to validate schema of database table '"users2"' for connection 'jdbc:postgresql://localhost:5433/postgres' with Connection to localhost:5433 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

[AFTER CHANGES]

Error message on the UI: Error encountered while configuring the stage: 'Error occurred while trying to get schema from database. Error message: 'Connection to localhost:5433 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.'. Error code: '0'. SQLState: '08001''

Is the new error message acceptable or should we revert the changes done to throw ProgramFailureException for validateSchema() method?

LGTM

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

database-commons/src/main/java/io/cdap/plugin/db/sink/AbstractDBSink.java

itsankit-google · 2025-06-26T08:36:46Z

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

  protected String getExternalDocumentationLink() {
-    return null;
+    return "https://en.wikipedia.org/wiki/SQLSTATE";


is this method still needed? can we remove it?

Not needed now. Removed. 7c8815d

itsankit-google · 2025-06-26T08:40:40Z

database-commons/src/main/java/io/cdap/plugin/db/action/AbstractDBArgumentSetter.java

+  protected String getExternalDocumentationLink() {
+    return "https://en.wikipedia.org/wiki/SQLSTATE";
+  }


is this method still needed?

No, not needed now. Removed it. 7c8815d

itsankit-google · 2025-06-26T11:25:10Z

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

@@ -87,13 +86,17 @@ public abstract class AbstractDBSource<T extends PluginConfig & DatabaseSourceCo
                                                                Pattern.CASE_INSENSITIVE);
  private static final Pattern WHERE_CONDITIONS = Pattern.compile("\\s+where \\$conditions",
                                                                  Pattern.CASE_INSENSITIVE);
+  private final RetryPolicy<?> retryPolicy;



please remove empty line

Removed 0b57ce6

database-commons/src/main/java/io/cdap/plugin/db/RetryExceptions.java

itsankit-google · 2025-07-10T18:03:00Z

cloudsql-mysql-plugin/src/e2e-test/resources/errorMessage.properties

@@ -1,9 +1,9 @@
-errorMessageInvalidSourceDatabase=SQL error while getting query schema: Error: Unknown database 'invalidDatabase', SQLState: 42000, ErrorCode: 1049
+errorMessageInvalidSourceDatabase=errorMessage: SQL Exception occurred


I still see generic error messages which is bad.

The earlier error message is changed as per the new code, and hence as per the latest code updated the error message not include the dynamic/privacy data present in error msgs here it contains.

As suggested, updated the error message.

itsankit-google

can we please add Open and Capture Logs step to these scenarios :

database-plugins/mssql-plugin/src/e2e-test/features/mssql/mssql source/RunTimeWithMacros.feature

Line 217 in e5c5794

Scenario: Verify pipeline failure message in logs when user provides invalid Table name in importQuery of plugin with Macros
database-plugins/mssql-plugin/src/e2e-test/features/mssql/mssql source/RunTimeWithMacros.feature

Line 258 in e5c5794

Scenario: Verify pipeline failure message in logs when user provides invalid Credentials for connection with Macros
database-plugins/cloudsql-postgresql-plugin/src/e2e-test/features/source/RunTime.feature

Line 114 in e5c5794

Scenario: To verify pipeline failure message in logs when an invalid bounding query is provided
database-plugins/cloudsql-postgresql-plugin/src/e2e-test/features/source/RunTime.feature

Line 156 in e5c5794

Scenario: To verify the pipeline fails while preview with invalid bounding query setting the split-By field

itsankit-google · 2025-07-10T18:24:20Z

database-commons/src/main/java/io/cdap/plugin/util/RetryUtils.java

+            } else if (cause instanceof RuntimeException) {
+                return (RuntimeException) cause;
+            } else if (cause instanceof Error) {
+                return new RuntimeException("Failsafe wrapped an Error", cause);


return new RuntimeException("Operation failed with error", cause);

Updated 7977d45

itsankit-google · 2025-07-10T18:25:44Z

database-commons/src/main/java/io/cdap/plugin/util/RetryUtils.java

+            } else if (cause instanceof Error) {
+                return new RuntimeException("Failsafe wrapped an Error", cause);
+            } else {
+                return new RuntimeException("Failsafe wrapped a non-runtime exception", cause);


return new RuntimeException("Operation failed", cause);

Updated 7977d45

itsmekumari · 2025-07-11T05:50:48Z

can we please add Open and Capture Logs step to these scenarios :

database-plugins/mssql-plugin/src/e2e-test/features/mssql/mssql source/RunTimeWithMacros.feature

Line 217 in e5c5794

Scenario: Verify pipeline failure message in logs when user provides invalid Table name in importQuery of plugin with Macros

database-plugins/mssql-plugin/src/e2e-test/features/mssql/mssql source/RunTimeWithMacros.feature

Line 258 in e5c5794

Scenario: Verify pipeline failure message in logs when user provides invalid Credentials for connection with Macros

database-plugins/cloudsql-postgresql-plugin/src/e2e-test/features/source/RunTime.feature

Line 114 in e5c5794

Scenario: To verify pipeline failure message in logs when an invalid bounding query is provided

database-plugins/cloudsql-postgresql-plugin/src/e2e-test/features/source/RunTime.feature

Line 156 in e5c5794

Scenario: To verify the pipeline fails while preview with invalid bounding query setting the split-By field

Can we address these changes in other PR? We will create a new ticket to address these changes. Please confirm.

itsankit-google · 2025-07-11T06:35:50Z

Can we address these changes in other PR? We will create a new ticket to address these changes. Please confirm.

I am not sure how to verify the e2e changes then?

itsmekumari · 2025-07-14T05:01:49Z

Can we address these changes in other PR? We will create a new ticket to address these changes. Please confirm.

I am not sure how to verify the e2e changes then?

Changes done.

database-commons/src/main/java/io/cdap/plugin/util/RetryUtils.java

sgarg-CS · 2025-07-18T05:01:56Z

hi @itsankit-google, Please let me know if there are any new review comments to be addressed. If not, can we approve it?

itsankit-google · 2025-08-07T14:47:59Z

database-commons/src/main/java/io/cdap/plugin/db/sink/AbstractDBSink.java

+        connection, String.format("SELECT %s FROM %s WHERE 1 = 0", dbColumns, fullyQualifiedTableName),
+          getErrorDetailsProvider());
+           ResultSet rs = RetryUtils.executeQueryWithRetry((RetryPolicy<ResultSet>) retryPolicy, pStmt,
+             getErrorDetailsProvider())) {
        getFieldsValidator().validateFields(inputSchema, rs, collector);
      }
    } catch (SQLException e) {


I don't think this catch is needed now?

It's needed to handle the SQLException thrown from the auto-closeable resource used in the try with resources block.

Both the places where ever we are connecting to SQL driver, I can see RetryUtils getting used, when will it go in this cache clause?

It might go.

As we're using try-with-resources, this will auto close the resources being used by implicitly calling close() method on each resource in the reverse order of declaration.
Since close() can throw SQLException, the compiler enforces that we either:

catch it, or

declare it in the method signature

Furthermore, we also have methods such as getMetaData(), getTables() and next() that can possibly throw SQLException

Can we add sqlState and errorCode in the error message of catch block as well?

itsankit-google · 2025-08-07T16:02:42Z

database-commons/src/main/java/io/cdap/plugin/db/sink/AbstractDBSink.java

          inferredFields.addAll(getSchemaReader().getSchemaFields(rs));
        }
-      } catch (SQLException e) {


we should also add it here then?

Otherwise it will fallback to lower one which will show incorrect error message

psainics force-pushed the patch/plugin-1823 branch from 7fcb0a0 to 9da75b8 Compare May 19, 2025 09:01

psainics added the build label May 19, 2025

itsmekumari mentioned this pull request May 22, 2025

error message changes #598

Merged

sgarg-CS force-pushed the patch/plugin-1823 branch from 9da75b8 to ac813f0 Compare May 26, 2025 05:28

sgarg-CS added build and removed build labels May 27, 2025

sgarg-CS requested a review from itsankit-google May 30, 2025 04:56

itsankit-google requested changes May 30, 2025

View reviewed changes

itsankit-google reviewed Jun 2, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/sink/AbstractDBSink.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 2, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/sink/AbstractDBSink.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 2, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 2, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 2, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/ConnectionConfig.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 2, 2025

View reviewed changes

cloudsql-mysql-plugin/src/test/java/io/cdap/plugin/cloudsql/mysql/CloudSQLMySQLSinkTest.java Show resolved Hide resolved

itsankit-google reviewed Jun 2, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/connector/AbstractDBSpecificConnector.java Outdated Show resolved Hide resolved

itsankit-google requested changes Jun 2, 2025

View reviewed changes

itsankit-google reviewed Jun 2, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 3, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/util/RetryUtils.java Show resolved Hide resolved

itsankit-google reviewed Jun 3, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 3, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/util/RetryUtils.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 4, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/connector/AbstractDBConnectorConfig.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 4, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/sink/AbstractDBSink.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 4, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 4, 2025

View reviewed changes

itsankit-google reviewed Jun 25, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 25, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/sink/AbstractDBSink.java Outdated Show resolved Hide resolved

itsankit-google reviewed Jun 26, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/RetryExceptions.java Show resolved Hide resolved

itsmekumari force-pushed the patch/plugin-1823 branch 3 times, most recently from c6de31b to 26efc64 Compare July 7, 2025 10:26

sgarg-CS mentioned this pull request Jul 8, 2025

[PLUGIN-1823] Retrying all SQLTransientExceptions #600

Closed

itsmekumari force-pushed the patch/plugin-1823 branch from 0e9b88e to 2284f2d Compare July 10, 2025 04:57

itsankit-google reviewed Jul 10, 2025

View reviewed changes

itsmekumari force-pushed the patch/plugin-1823 branch from cbab045 to 1c0bfe5 Compare July 11, 2025 12:42

sgarg-CS force-pushed the patch/plugin-1823 branch from 7977d45 to 40ff60a Compare July 14, 2025 06:07

itsankit-google reviewed Jul 16, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/util/RetryUtils.java Outdated Show resolved Hide resolved

sgarg-CS force-pushed the patch/plugin-1823 branch from 7e16744 to 3f332d1 Compare July 17, 2025 05:46

sgarg-CS force-pushed the patch/plugin-1823 branch 2 times, most recently from 148f5a7 to 766132b Compare August 7, 2025 07:38

PLUGIN-1823: Retrying all SQLTransientExceptions

40264b9

sgarg-CS force-pushed the patch/plugin-1823 branch from 766132b to 40264b9 Compare August 7, 2025 07:39

itsankit-google approved these changes Aug 7, 2025

View reviewed changes

itsankit-google reviewed Aug 7, 2025

View reviewed changes

		@@ -1,9 +1,9 @@
		errorMessageInvalidSourceDatabase=SQL error while getting query schema: Error: Unknown database 'invalidDatabase', SQLState: 42000, ErrorCode: 1049
		errorMessageInvalidSourceDatabase=errorMessage: SQL Exception occurred

PLUGIN-1823: Retrying all SQLTransientExceptions #597

Are you sure you want to change the base?

PLUGIN-1823: Retrying all SQLTransientExceptions #597

Conversation

sgarg-CS commented May 16, 2025 • edited by jira bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

google-cla bot commented May 16, 2025

Uh oh!

itsankit-google left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

itsankit-google commented May 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

itsankit-google left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

itsankit-google left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sgarg-CS commented Jun 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

itsankit-google left a comment

Choose a reason for hiding this comment

Uh oh!

sgarg-CS commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

itsankit-google commented Jun 5, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itsankit-google left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

sgarg-CS commented May 16, 2025 •

edited by jira bot

Loading

itsankit-google left a comment •

edited

Loading

sgarg-CS commented Jun 5, 2025 •

edited

Loading

itsankit-google left a comment •

edited

Loading

itsankit-google Aug 7, 2025 •

edited

Loading