Skip to content

[SPARK-53125][TEST] RemoteSparkSession prints whole spark-submit command #51846

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

pan3793
Copy link
Member

@pan3793 pan3793 commented Aug 5, 2025

What changes were proposed in this pull request?

Make RemoteSparkSession print the whole spark-submit command in debug mode, which helps the developers to understand the SPARK_HOME, classpath, log output, etc. of the testing connect server process.

Why are the changes needed?

Improve the debug message.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

export SPARK_DEBUG_SC_JVM_CLIENT=true
sbt:spark-connect-client-jvm> testOnly *ClientE2ETestSuite -- -z "throw SparkException with large cause exception"
[info] ClientE2ETestSuite:
Starting the Spark Connect Server...
Using jar: /Users/chengpan/Projects/apache-spark/sql/connect/server/target/scala-2.13/spark-connect-assembly-4.1.0-SNAPSHOT.jar
Using jar: /Users/chengpan/Projects/apache-spark/sql/catalyst/target/scala-2.13/spark-catalyst_2.13-4.1.0-SNAPSHOT-tests.jar
/Users/chengpan/Projects/apache-spark/bin/spark-submit \
    --driver-class-path /Users/chengpan/Projects/apache-spark/sql/connect/server/target/scala-2.13/spark-connect-assembly-4.1.0-SNAPSHOT.jar \
    --class org.apache.spark.sql.connect.SimpleSparkConnectService \
    --jars /Users/chengpan/Projects/apache-spark/sql/catalyst/target/scala-2.13/spark-catalyst_2.13-4.1.0-SNAPSHOT-tests.jar \
    --conf spark.connect.grpc.binding.port=15707 \
    --conf spark.sql.catalog.testcat=org.apache.spark.sql.connector.catalog.InMemoryTableCatalog \
    --conf spark.sql.catalogImplementation=hive \
    --conf spark.connect.execute.reattachable.senderMaxStreamDuration=1s \
    --conf spark.connect.execute.reattachable.senderMaxStreamSize=123 \
    --conf spark.connect.grpc.arrow.maxBatchSize=10485760 \
    --conf spark.ui.enabled=false \
    --conf spark.driver.extraJavaOptions=-Dlog4j.configurationFile=/Users/chengpan/Projects/apache-spark/sql/connect/client/jvm/src/test/resources/log4j2.properties /Users/chengpan/Projects/apache-spark/sql/connect/server/target/scala-2.13/spark-connect-assembly-4.1.0-SNAPSHOT.jar
...

Was this patch authored or co-authored using generative AI tooling?

No.

@pan3793
Copy link
Member Author

pan3793 commented Aug 6, 2025

cc @HyukjinKwon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant