[SPARK-45720][BUILD][DSTREAM][KINESIS] Upgrade KCL to 2.7.1 and remove AWS SDK for Java 1.x dependency #53256
+561
−373
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Upgrade codes related to IAM, DDB and Kinesis clients Shade SDKv2 dependencies Use Spark's default protobuf version in the kinesis-asl-assembly module Import mbknor-jackson-jsonschema_2.13 for glue-schema-registry Jira: https://issues.apache.org/jira/browse/SPARK-45720
Fix jaxb version
sarutak
commented
Nov 28, 2025
| <!-- Should be consistent with SparkBuild.scala and docs --> | ||
| <avro.version>1.12.1</avro.version> | ||
| <aws.kinesis.client.version>1.15.3</aws.kinesis.client.version> | ||
| <aws.kinesis.client.version>2.7.1</aws.kinesis.client.version> |
Member
Author
There was a problem hiding this comment.
2.7.1 is not the latest version but this version works with AWS SDK for Java 2.29.52 Spark currently depends on.
2.7.2 depends on the SDK 2.33.0 and noticed it doesn't works with 2.29.52.
sarutak
commented
Nov 28, 2025
| val testData3 = 21 to 30 | ||
|
|
||
| eventually(timeout(1.minute), interval(10.seconds)) { | ||
| eventually(timeout(2.minute), interval(10.seconds)) { |
Member
Author
There was a problem hiding this comment.
Scheduler takes about 1 min by default for initialization so timeout is increased here.
sarutak
commented
Nov 28, 2025
| * @param shardEndedInput provides access to a checkpointer method for completing processing of | ||
| * the shard. | ||
| */ | ||
| override def shutdown( |
Member
Author
There was a problem hiding this comment.
shutdown is divided into three functions for the corresponding shutdown reasons.
- TERMINATE => shardEnded
- REQUESTED => shutdownRequested
- ZOMBIE => leaseLost
In the current code, REQUESTED is not cared but in this case checkpoint can be done. So, checkpoint is done in shutdownRequested.
Member
|
Thank you so much, @sarutak . |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR proposes to upgrade KCL to 2.7.1 based on @junyuc25 's PR with some updates.
By upgrading KCL, we can remove AWS SDK for Java 1.x dependency.
Why are the changes needed?
Does this PR introduce any user-facing change?
Expect the behavior is not changed.
How was this patch tested?
Confirmed all kinesis tests passed with the following commands.
Also confirmed existing examples work.
Was this patch authored or co-authored using generative AI tooling?
No.