Add `pathStyleAccess` to AwsStorageConfigInfo #2012

dimas-b · 2025-07-08T23:46:48Z

This change allows configuring the "path-style" access mode in S3 clients (both in Polaris Servers and Iceberg REST Catalog API clients).

This change is applicable both to AWS storage and to non-AWS S3-compatible storage (#1530).

The Management API change is backward-compatible.

Includes a fix to AWS config object round-trip via generated API classes.

Dev ML discussion: https://lists.apache.org/thread/oxy2yd4mbq06zr4z57lfptgy95c3lyhb

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java

eric-maynard · 2025-07-09T03:03:10Z

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java

  }

  public AwsStorageConfigurationInfo(
      @Nonnull StorageType storageType,
      @Nonnull List<String> allowedLocations,
      @Nonnull String roleARN,
      @Nullable String region) {
-    this(storageType, allowedLocations, roleARN, null, region, null, null);
+    this(storageType, allowedLocations, roleARN, null, region, null, null, false);


We could use the default here, or just leave it null to leverage the default used in orElse. Better not to proliferate the constants around like this.

changed to use PATH_STYLE_ACCESS_DEFAULT

Now that we have a default at the API level, is this actually nullable?

I had to remove the API default.

eric-maynard · 2025-07-09T03:04:06Z

spec/polaris-management-service.yml

+              type: boolean
+              description: >- 
+                Whether S3 requests to files in this catalog should use 'path-style addressing for buckets'.
+                Default: false.


I believe OpenAPI actually supports default values, maybe we could just use that?

Added default: false, changed example to true.

Actually I had to roll this back. Using Open API defaults breaks strict backward compatibility because is causes the attribute with a default value to be always returned to clients (even when not in use) - CI caught this.

What exactly was the failure in CI? The client would fail to parse a response because it didn't recognize the property?

CatalogSerializationTest detected a change in the JSON text (when the new properties are not set)

actually, never mind, I'll expand this PR a bit :)

updated. PTAL.

Thanks, the changes look better, but my concern about the default remains

TBH, I do not understand the whole discussion about "is this a breaking change".

As Polaris is primarily an Iceberg REST catalog, we have to let JSON parsing ignore unknown properties, because that is a requirement from Iceberg REST. There is also exactly one JSON parser (ObjectMapper) for all endpoints. The situation is unfortunate but a consequence of generating the OpenAPI code. Jackson itself allows much more fine grained control over how types are (de)serialized and allows back/forwards compatibility targeting functionality like @JsonView - but that's not usable w/ the code generation approach.

So fact is that unknown properties in JSON objects in any Polaris REST payload have to be ignored. As that fact leaks to all Polaris APIs, it also leaks to clients. This means, that a new property is not a breaking change. If clients fail on unknown properties, it's a bug in those client.

@eric-maynard : the new test shows AWS config data roundtrips via API with and without (defaults) values for the new properties.

If the new properties are not set, clients will not see them in JSON outputs of the REST API endpoints. So, this change does not break existing installations and old clients as long as Polaris users do not use the new properties. If users do reconfigure catalogs to use the new properties, it's the user's responsibility to also update clients to be compatible with the catalog JSON configuration (e.g. use new CLI).

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java

singhpk234

LGTM !

dimas-b · 2025-07-09T14:44:58Z

I had to remove one commit (discussed in the API yaml comment thread). PTAL.

eric-maynard

Let's clarify if this is a breaking change (does it need to target 2.0? Go into API version 2?) and plan accordingly before merging.

snazy

LGTM

This change allows configuring the "path-style" access mode in S3 clients (both in Polaris Servers and Iceberg REST Catalog API clients). This change is applicable both to AWS storage and to non-AWS S3-compatible storage (apache#1530). The Management API change is backward-compatible.

dimas-b · 2025-07-14T17:10:12Z

@eric-maynard : I believe I addressed your backward-compatibility concern in the latest push. Could you review again?

dimas-b requested review from snazy and eric-maynard July 8, 2025 23:46

github-project-automation bot added this to Basic Kanban Board Jul 8, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jul 8, 2025

eric-maynard reviewed Jul 9, 2025

View reviewed changes

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java Outdated Show resolved Hide resolved

eric-maynard reviewed Jul 9, 2025

View reviewed changes

snazy previously approved these changes Jul 9, 2025

View reviewed changes

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java Outdated Show resolved Hide resolved

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jul 9, 2025

dimas-b dismissed snazy’s stale review via b889c1e July 9, 2025 14:28

singhpk234 reviewed Jul 9, 2025

View reviewed changes

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java Outdated Show resolved Hide resolved

singhpk234 previously approved these changes Jul 9, 2025

View reviewed changes

dimas-b dismissed singhpk234’s stale review via 780f36b July 9, 2025 14:42

dimas-b force-pushed the s3-path-style branch from b889c1e to 780f36b Compare July 9, 2025 14:42

eric-maynard requested changes Jul 10, 2025

View reviewed changes

github-project-automation bot moved this from Ready to merge to PRs In Progress in Basic Kanban Board Jul 10, 2025

snazy previously approved these changes Jul 10, 2025

View reviewed changes

dimas-b added 3 commits July 10, 2025 18:03

review: rename to getPathStyleAccess

c86f3e9

review: PATH_STYLE_ACCESS_DEFAULT, ternary

78cdf5b

dimas-b dismissed snazy’s stale review via 090b9b6 July 10, 2025 22:50

dimas-b force-pushed the s3-path-style branch from 780f36b to 090b9b6 Compare July 10, 2025 22:50

dimas-b requested review from eric-maynard, snazy and singhpk234 July 10, 2025 22:51

fix config round trip, add tests

09f5d77

dimas-b force-pushed the s3-path-style branch from 090b9b6 to 09f5d77 Compare July 10, 2025 23:10

Add pathStyleAccess to AwsStorageConfigInfo #2012

Are you sure you want to change the base?

Add pathStyleAccess to AwsStorageConfigInfo #2012

Conversation

dimas-b commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

dimas-b commented Jul 9, 2025

Uh oh!

eric-maynard left a comment

Choose a reason for hiding this comment

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

dimas-b commented Jul 14, 2025

Uh oh!

Uh oh!

Add `pathStyleAccess` to AwsStorageConfigInfo #2012

Add `pathStyleAccess` to AwsStorageConfigInfo #2012

dimas-b commented Jul 8, 2025 •

edited

Loading