Add Spark Connect support for Spark datasets

[zhengruifeng]

What does this PR do?

Adds Spark Connect support to Dataset.from_spark and IterableDataset.from_spark by replacing client-side RDD and SparkContext calls with DataFrame APIs supported by Spark Connect.

• discovers partitions with spark_partition_id() for streaming datasets
• runs the shared-cache probe with mapInArrow
• distributes shard renames with createDataFrame and mapInArrow
• adds coverage for partition discovery, empty DataFrames, the remote-cache probe, and multi-shard output

Testing

• ruff check src/datasets/packaged_modules/spark/spark.py tests/packaged_modules/test_spark.py tests/test_arrow_dataset.py
• ruff format --check src/datasets/packaged_modules/spark/spark.py tests/packaged_modules/test_spark.py tests/test_arrow_dataset.py
• PySpark 4.2 classic and Spark Connect smoke tests for partition discovery, worker cache probing, and distributed shard renaming