HADOOP-19343: Include hadoop-gcp in distro. #7877

cnauroth · 2025-08-17T22:28:22Z

Description of PR

Include hadoop-gcp in hadoop-tools-dist, so that it makes it into the final release tarball.

How was this patch tested?

mvn -Pdist -Dtar -DskipTests clean package

tar tf hadoop-dist/target/hadoop-3.5.0-SNAPSHOT.tar.gz hadoop-3.5.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-gcp-3.5.0-SNAPSHOT.jar
hadoop-3.5.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-gcp-3.5.0-SNAPSHOT.jar

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

cnauroth · 2025-08-17T22:28:50Z

This will need a rebase after #7874 gets committed.

Closes apache#7877

hadoop-yetus · 2025-08-18T00:58:52Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ HADOOP-19343 Compile Tests _
+0 🆗	mvndep	10m 51s		Maven dependency ordering for branch
+1 💚	mvninstall	41m 18s		HADOOP-19343 passed
+1 💚	compile	1m 39s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 22s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	mvnsite	1m 1s		HADOOP-19343 passed
+1 💚	javadoc	0m 59s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 52s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	93m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 5s		Maven dependency ordering for patch
+1 💚	mvninstall	10m 35s		the patch passed
+1 💚	compile	1m 29s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 29s		the patch passed
+1 💚	compile	1m 15s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	1m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 43s		the patch passed
+1 💚	javadoc	0m 39s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 38s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	37m 40s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 29s		hadoop-gcp in the patch passed.
+1 💚	unit	0m 22s		hadoop-tools-dist in the patch passed.
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		149m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7877/1/artifact/out/Dockerfile
GITHUB PR	#7877
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux c86033703477 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HADOOP-19343 / `793a50d`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7877/1/testReport/
Max. process+thread count	556 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-gcp hadoop-tools/hadoop-tools-dist U: hadoop-tools
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7877/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-08-18T01:11:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	14m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ HADOOP-19343 Compile Tests _
+0 🆗	mvndep	10m 32s		Maven dependency ordering for branch
+1 💚	mvninstall	40m 15s		HADOOP-19343 passed
+1 💚	compile	1m 37s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 20s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	mvnsite	1m 1s		HADOOP-19343 passed
+1 💚	javadoc	0m 58s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 54s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	91m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 5s		Maven dependency ordering for patch
+1 💚	mvninstall	10m 32s		the patch passed
+1 💚	compile	1m 30s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 30s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	1m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 42s		the patch passed
+1 💚	javadoc	0m 37s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 38s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	37m 9s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 29s		hadoop-gcp in the patch passed.
+1 💚	unit	0m 22s		hadoop-tools-dist in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		160m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7877/2/artifact/out/Dockerfile
GITHUB PR	#7877
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 686c31027f59 5.15.0-140-generic #150-Ubuntu SMP Sat Apr 12 06:00:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HADOOP-19343 / `793a50d`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7877/2/testReport/
Max. process+thread count	704 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-gcp hadoop-tools/hadoop-tools-dist U: hadoop-tools
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7877/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

arunkumarchacko · 2025-08-19T02:43:43Z

hadoop-tools/hadoop-gcp/pom.xml

@@ -351,7 +351,6 @@
                  </shadedPattern>
                </relocation>
              </relocations>
-              <shadedArtifactAttached>true</shadedArtifactAttached>


Why are we removing this?

We have two options:

shadedArtifactAttached=true means a separate shaded jar artifact will be attached to the build. Clients can depend on it by specifying <classifier>shaded</classifier> in the dependency declaration.

shadedArtifactAttached=false (the default, hence removed in this patch) means the shaded jar completely replaces the published artifact. I think this is what we want for hadoop-gcp, given the tight coupling to specific shaded versions of things like Guava and gRPC. There wouldn't be much value in the non-shaded artifacts.

ayushtkn

LGTM

Closes #7877 Signed-off-by: Ayush Saxena <[email protected]> Reviewed-by: Arunkumar Chacko <[email protected]>

cnauroth · 2025-08-25T21:10:04Z

I merged this to the feature branch. Thank you for the reviews, @arunkumarchacko and @ayushtkn .

Closes #7877 Signed-off-by: Ayush Saxena <[email protected]> Reviewed-by: Arunkumar Chacko <[email protected]>

Closes apache#7877 Signed-off-by: Ayush Saxena <[email protected]> Reviewed-by: Arunkumar Chacko <[email protected]>

github-actions bot added build trunk TOOLS labels Aug 17, 2025

HADOOP-19343: Include hadoop-gcp in distro.

793a50d

Closes apache#7877

cnauroth force-pushed the HADOOP-19343-gcp-dist branch from a5bf1f6 to 793a50d Compare August 17, 2025 22:29

arunkumarchacko reviewed Aug 19, 2025

View reviewed changes

arunkumarchacko approved these changes Aug 19, 2025

View reviewed changes

cnauroth requested a review from slfan1989 August 25, 2025 15:35

ayushtkn approved these changes Aug 25, 2025

View reviewed changes

cnauroth added a commit that referenced this pull request Aug 25, 2025

HADOOP-19343: Include hadoop-gcp in distro.

a729021

Closes #7877 Signed-off-by: Ayush Saxena <[email protected]> Reviewed-by: Arunkumar Chacko <[email protected]>

cnauroth closed this Aug 25, 2025

cnauroth added a commit that referenced this pull request Aug 25, 2025

HADOOP-19343: Include hadoop-gcp in distro.

d90e691

Closes #7877 Signed-off-by: Ayush Saxena <[email protected]> Reviewed-by: Arunkumar Chacko <[email protected]>

cnauroth mentioned this pull request Aug 25, 2025

HADOOP-19343: Manage hadoop-gcp Guava version directly in its pom.xml. #7883

Closed

4 tasks

cnauroth added a commit to cnauroth/hadoop that referenced this pull request Aug 29, 2025

HADOOP-19343: Include hadoop-gcp in distro.

5538e9c

Closes apache#7877 Signed-off-by: Ayush Saxena <[email protected]> Reviewed-by: Arunkumar Chacko <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19343: Include hadoop-gcp in distro. #7877

HADOOP-19343: Include hadoop-gcp in distro. #7877

cnauroth commented Aug 17, 2025

Uh oh!

cnauroth commented Aug 17, 2025

Uh oh!

hadoop-yetus commented Aug 18, 2025

Uh oh!

hadoop-yetus commented Aug 18, 2025

Uh oh!

arunkumarchacko Aug 19, 2025

Uh oh!

cnauroth Aug 19, 2025

Uh oh!

ayushtkn left a comment

Uh oh!

cnauroth commented Aug 25, 2025

Uh oh!

Uh oh!

HADOOP-19343: Include hadoop-gcp in distro. #7877

HADOOP-19343: Include hadoop-gcp in distro. #7877

Conversation

cnauroth commented Aug 17, 2025

Description of PR

How was this patch tested?

For code changes:

Uh oh!

cnauroth commented Aug 17, 2025

Uh oh!

hadoop-yetus commented Aug 18, 2025

Uh oh!

hadoop-yetus commented Aug 18, 2025

Uh oh!

arunkumarchacko Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

cnauroth Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

cnauroth commented Aug 25, 2025

Uh oh!

Uh oh!