Extensible pagination token implementation #1938

snazy · 2025-06-25T10:30:03Z

Based on #1838, following up on #1555

Allows multiple implementations of Token referencing the "next page", encapsulated in PageToken. No changes to polaris-core needed to add custom Token implementations.
Extensible to (later) support (cryptographic) signatures to prevent tampered page-token
Refactor pagination code to delineate API-level page tokens and internal "pointers to data"
Requests deal with the "previous" token, user-provided page size (optional) and the previous request's page size.
Concentrate the logic of combining page size requests and previous tokens in PageTokenUtil
PageToken subclasses are no longer necessary.
Serialzation of PageToken uses Jackson serialization (smile format)

Since no (metastore level) implementation handling pagination existed before, no backwards compatibility is needed.

snazy · 2025-06-25T16:01:30Z

This approach also works for NoSQL #1189 (last commit in this branch)

eric-maynard · 2025-06-25T16:25:18Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/dao/entity/EntitiesResult.java

-  }
-
-  @JsonCreator
-  private EntitiesResult(


If we can avoid it, I think it's better not to change this. This will be a breaking change for any persistence implementation depending on wire-compatibility of the result.

cc @dennishuo for visibility as well

It is a private constructor... @eric-maynard @dennishuo : could you give some more details about possible compatibility issues? What's the affected workflow?

I do not see any roundtrips through JSON for EntitiesResult in Apache Polaris code. If this is a use case downstream, it would be nice to have tests in this repo to assert the expected behaviour.

That said, I think this specific change is not critical to pagination. EntitiesResult is not involved handling paginated data. So, I suppose this change can be rolled back.

I do not see a "wire" either.
BTW: The same change wasn't considered an issue in 1838.

eric-maynard · 2025-06-25T16:25:59Z

...is-core/src/main/java/org/apache/polaris/core/persistence/dao/entity/ListEntitiesResult.java

  }

-  @JsonCreator
-  private ListEntitiesResult(


ditto the comment on wire-compatibility with previous versions

@eric-maynard : It's good that you're able to identify this change as a regression, but what if you're not available ? :) WDYT about encoding this expectation in a unit test?

eric-maynard · 2025-06-25T16:26:18Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdToken.java

+public interface EntityIdToken extends Token {
+  String ID = "e";
+
+  @JsonProperty("i")


Why would we use the property name i?

I believe short names are meant to reduce the side of the token on the wire.

Yes, as explained in Token

eric-maynard · 2025-06-25T16:26:32Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdToken.java

+  long entityId();
+
+  @Override
+  default String getT() {


The reason is explained as doc in Token

eric-maynard · 2025-06-25T16:27:10Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdToken.java

+@JsonSerialize(as = ImmutableEntityIdToken.class)
+@JsonDeserialize(as = ImmutableEntityIdToken.class)
+public interface EntityIdToken extends Token {
+  String ID = "e";


Is this meant to be short for EntityId? Why not use an enum?

That contradicts the intent of being extensible.

eric-maynard · 2025-06-25T16:28:23Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java

+
+      // Signal "no more data" if the number of items is less than the requested page size or if
+      // there is no more data.
+      if (data.size() < limit || !it.hasNext()) {


It should be expected that the iterator has no more items if the limit has been pushed down into whatever provided the items, no?

Good point! It also looks like we do not push limits down to the database.

I propose to remove !it.hasNext() for the sake of simplicity, accept possible empty last pages, and work on exact page boundaries later.

There is a test for this - and it passes

I wonder whether the test passes because limits are not pushed down to Persistence queries anywhere 🤔

eric-maynard · 2025-06-25T16:28:54Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java

+   * servicing the request for the next page of related data.
+   */
+  public @Nullable String encodedResponseToken() {
+    return PageTokenUtil.encodePageToken(request, nextToken);


Same comments from the previous PR -- encoding/decoding a given PageToken clearly seems to be within the purview of a given PageToken implementation

There is only one PageToken.

eric-maynard · 2025-06-25T16:29:09Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java

+@PolarisImmutable
+@JsonSerialize(as = ImmutablePageToken.class)
+@JsonDeserialize(as = ImmutablePageToken.class)


Would rather avoid these if possible

eric-maynard · 2025-06-25T16:29:19Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java

-    return build(null, pageSize);
-  }
+  /** The requested page size (optional). */
+  @JsonProperty("p")


ditto the comment on i -- this is not a good property name.

eric-maynard · 2025-06-25T16:29:33Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java

   */
-  protected abstract PageToken updated(List<?> newData);
+  @JsonProperty("v")


eric-maynard · 2025-06-25T16:29:57Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java

   */
-  protected abstract PageToken updated(List<?> newData);
+  @JsonProperty("v")
+  Optional<Token> value();


What is value? This seems a bit abstract.

eric-maynard · 2025-06-25T16:30:17Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java

-    }
+  /** Convenience for {@code pageSize().isPresent()}. */
+  default boolean paginationRequested() {
+    return pageSize().isPresent();


It's possible per the spec to request pagination by providing a page token without page size. This logic is not correct.

The logic looks correct to me. Propagating page size from the previous token in handled in PageTokenUtil.decodePageRequest()

eric-maynard · 2025-06-25T16:30:46Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java

+
+  /** Represents a non-paginated request. */
+  static PageToken readEverything() {
+    return PageTokenUtil.READ_EVERYTHING;


Is READ_EVERYTHING a Util? Seems like it would belong in PageToken

eric-maynard · 2025-06-25T16:32:10Z

Thanks for taking a crack at this. I appreciate the extra extensibility compared to #1838, but considering this diff is twice the size of #1555, I'm not sure there's enough functionality here to justify all the changes. I took a quick pass for now, but will try to circle back and review the rest of the PR in more detail later.

eric-maynard · 2025-06-25T16:33:28Z

...a/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java

+    data = data.sorted(Comparator.comparingLong(PolarisEntityCore::getId)).filter(tokenFilter);
+
+    data = data.filter(entityFilter);


Why break this into two calls?

(I think it came from my PR) No particular reason. Would you prefer a chained call?

dimas-b

Changes LGTM 👍

dimas-b · 2025-06-26T03:09:08Z

...t/resources/META-INF/services/org.apache.polaris.core.persistence.pagination.Token$TokenType

Token$TokenType looks awkward as a file name ($ and double Token)... WDYT about making TokenType a top-level class?

It's the class name.
I'd really prefer keeping these very related things (Token + TokenType) together - you have to implement both.

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java

snazy · 2025-06-26T06:34:34Z

This change enables pagination also for #1189, whereas #1555 does not.

Based on apache#1838, following up on apache#1555 * Allows multiple implementations of `Token` referencing the "next page", encapsulated in `PageToken`. No changes to `polaris-core` needed to add custom `Token` implementations. * Extensible to (later) support (cryptographic) signatures to prevent tampered page-token * Refactor pagination code to delineate API-level page tokens and internal "pointers to data" * Requests deal with the "previous" token, user-provided page size (optional) and the previous request's page size. * Concentrate the logic of combining page size requests and previous tokens in `PageTokenUtil` * `PageToken` subclasses are no longer necessary. * Serialzation of `PageToken` uses Jackson serialization (smile format) Since no (metastore level) implementation handling pagination existed before, no backwards compatibility is needed. Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: Eric Maynard <[email protected]>

github-project-automation bot added this to Basic Kanban Board Jun 25, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jun 25, 2025

snazy force-pushed the pagination-alt-flex branch from 12ab744 to 1b42120 Compare June 25, 2025 10:45

snazy requested a review from dimas-b June 25, 2025 13:16

dimas-b requested a review from eric-maynard June 25, 2025 16:25

eric-maynard reviewed Jun 25, 2025

View reviewed changes

dimas-b reviewed Jun 26, 2025

View reviewed changes

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageToken.java Show resolved Hide resolved

snazy force-pushed the pagination-alt-flex branch 3 times, most recently from 191d99d to 6cdb73c Compare July 1, 2025 04:56

snazy force-pushed the pagination-alt-flex branch 3 times, most recently from d072060 to 88ae32d Compare July 7, 2025 08:49

snazy force-pushed the pagination-alt-flex branch 2 times, most recently from 37eb5c3 to b472eef Compare July 11, 2025 10:41

snazy force-pushed the pagination-alt-flex branch from b472eef to edb7cc4 Compare July 12, 2025 12:05

		data = data.sorted(Comparator.comparingLong(PolarisEntityCore::getId)).filter(tokenFilter);

		data = data.filter(entityFilter);

Extensible pagination token implementation #1938

Are you sure you want to change the base?

Extensible pagination token implementation #1938

Conversation

snazy commented Jun 25, 2025

Uh oh!

snazy commented Jun 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-maynard Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-maynard commented Jun 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Jun 26, 2025 •

edited

Loading

dimas-b Jun 26, 2025 •

edited

Loading

eric-maynard Jun 25, 2025 •

edited

Loading

dimas-b left a comment •

edited

Loading